2024-03-29T11:22:32.475 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:22:32.475 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --owner scheduled_amathuri@teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713 -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/orig.config.yaml 2024-03-29T11:22:32.511 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:22:32.580 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622713' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPiPMOp+vgiEZZNa8LPYohHLQRxJY9R6LGA1gZHjclHL8fx7cA2Mxw4cYFT8DwlUftEIvyJowLK2lvMRhM8DeK8= smithi203.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZDVBhzAjM1D8MQiaC92e7nFlm/phBSL2LVW0XsOysQ2+y1EDEUrkFPDQ5iXfuu8mbjppQok81+OL7WRQ6WzLs= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:22:32.580 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:22:32.581 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:22:32.581 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:22:32.583 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:22:32.606 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:22:32.606 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:22:32.606 INFO:teuthology.packaging:ref: None 2024-03-29T11:22:32.606 INFO:teuthology.packaging:tag: None 2024-03-29T11:22:32.606 INFO:teuthology.packaging:branch: main 2024-03-29T11:22:32.606 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:22:32.606 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:22:32.890 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:22:32.891 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:22:32.898 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:22:32.899 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:22:32.926 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:22:32.939 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:22:33.024 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:22:33.047 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:38.098177', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPiPMOp+vgiEZZNa8LPYohHLQRxJY9R6LGA1gZHjclHL8fx7cA2Mxw4cYFT8DwlUftEIvyJowLK2lvMRhM8DeK8='} 2024-03-29T11:22:33.069 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:38.100132', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZDVBhzAjM1D8MQiaC92e7nFlm/phBSL2LVW0XsOysQ2+y1EDEUrkFPDQ5iXfuu8mbjppQok81+OL7WRQ6WzLs='} 2024-03-29T11:22:33.069 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:22:33.076 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:22:33.076 INFO:teuthology.task.internal:roles: ubuntu@smithi203.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:22:33.076 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:22:33.162 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa7d107e0d0>, signals=[15]) 2024-03-29T11:22:33.163 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:22:33.169 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:22:33.169 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:22:33.171 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:33.260 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:22:33.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:22:33.349 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:22:33.358 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2024-03-29T11:22:33.375 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2024-03-29T11:22:33.375 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Stream" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="9" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="9" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:33.431 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:33.432 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:33.432 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:33.432 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2024-03-29T11:22:33.450 DEBUG:teuthology.orchestra.run.smithi203:> uname -m 2024-03-29T11:22:33.468 INFO:teuthology.orchestra.run.smithi203.stdout:x86_64 2024-03-29T11:22:33.468 DEBUG:teuthology.orchestra.run.smithi203:> cat /etc/os-release 2024-03-29T11:22:33.525 INFO:teuthology.orchestra.run.smithi203.stdout:NAME="CentOS Stream" 2024-03-29T11:22:33.525 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION="9" 2024-03-29T11:22:33.525 INFO:teuthology.orchestra.run.smithi203.stdout:ID="centos" 2024-03-29T11:22:33.525 INFO:teuthology.orchestra.run.smithi203.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:22:33.525 INFO:teuthology.orchestra.run.smithi203.stdout:VERSION_ID="9" 2024-03-29T11:22:33.526 INFO:teuthology.orchestra.run.smithi203.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:22:33.526 INFO:teuthology.orchestra.run.smithi203.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:22:33.526 INFO:teuthology.orchestra.run.smithi203.stdout:ANSI_COLOR="0;31" 2024-03-29T11:22:33.526 INFO:teuthology.orchestra.run.smithi203.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:22:33.526 INFO:teuthology.orchestra.run.smithi203.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:22:33.526 INFO:teuthology.orchestra.run.smithi203.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:22:33.526 INFO:teuthology.orchestra.run.smithi203.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:22:33.526 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:22:33.526 INFO:teuthology.orchestra.run.smithi203.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:22:33.526 INFO:teuthology.lock.ops:Updating smithi203.front.sepia.ceph.com on lock server 2024-03-29T11:22:33.550 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:22:33.559 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:22:33.568 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:22:33.568 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:33.570 DEBUG:teuthology.orchestra.run.smithi203:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:22:33.586 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:22:33.603 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:22:33.604 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:33.626 DEBUG:teuthology.orchestra.run.smithi203:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:22:33.728 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:22:33.779 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:22:33.792 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:22:33.793 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:22:33.793 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:33.793 DEBUG:teuthology.orchestra.run.smithi203:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:22:33.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:33.809 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-03-29T11:22:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:33.810 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-29T11:22:33.867 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:33.867 INFO:teuthology.task.kernel:Running kernel on smithi145: 5.14.0-361.el9.x86_64 2024-03-29T11:22:33.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2024-03-29T11:22:33.868 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:22:33.868 INFO:teuthology.task.kernel:Running kernel on smithi203: 5.14.0-361.el9.x86_64 2024-03-29T11:22:33.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum install -y kernel 2024-03-29T11:22:36.015 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-29T11:22:36.152 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T11:22:37.566 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-29T11:22:38.011 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 2.5 MB/s | 2.0 MB 00:00 2024-03-29T11:22:38.773 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-03-29T11:22:39.001 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-29T11:22:40.866 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-29T11:22:45.542 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.1 MB 00:01 2024-03-29T11:22:47.014 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 2.7 MB/s | 19 MB 00:06 2024-03-29T11:22:47.176 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-29T11:22:49.421 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 9.8 MB/s | 21 MB 00:02 2024-03-29T11:22:51.186 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 9.3 MB/s | 6.1 MB 00:00 2024-03-29T11:22:52.962 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-03-29T11:22:54.385 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-03-29T11:22:55.674 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 8.0 MB/s | 21 MB 00:02 2024-03-29T11:22:56.592 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:22:56.592 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:22:56.727 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:22:56.735 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:22:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:22:56.735 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:22:56.735 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:22:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:22:56.735 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:22:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:22:56.735 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:22:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:22:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:22:56.736 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:22:56.736 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:22:56.736 INFO:teuthology.orchestra.run.smithi145.stdout:Install 4 Packages 2024-03-29T11:22:56.736 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:22:56.737 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 76 M 2024-03-29T11:22:56.737 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 123 M 2024-03-29T11:22:56.737 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:22:56.911 INFO:teuthology.orchestra.run.smithi145.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-29T11:22:57.653 INFO:teuthology.orchestra.run.smithi145.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 16 MB/s | 15 MB 00:00 2024-03-29T11:22:58.503 INFO:teuthology.orchestra.run.smithi145.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 19 MB/s | 34 MB 00:01 2024-03-29T11:22:58.754 INFO:teuthology.orchestra.run.smithi145.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 15 MB/s | 28 MB 00:01 2024-03-29T11:22:58.757 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:22:58.757 INFO:teuthology.orchestra.run.smithi145.stdout:Total 38 MB/s | 76 MB 00:02 2024-03-29T11:22:58.757 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:22:59.128 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:22:59.128 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:22:59.986 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:22:59.986 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:23:00.716 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T11:23:02.283 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:23:02.857 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:02.954 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:03.023 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:23:03.023 INFO:teuthology.orchestra.run.smithi203.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:23:03.120 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:23:03.131 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:23:03.132 INFO:teuthology.orchestra.run.smithi203.stdout:Install 4 Packages 2024-03-29T11:23:03.133 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:23:03.133 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 76 M 2024-03-29T11:23:03.133 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 123 M 2024-03-29T11:23:03.133 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:23:03.351 INFO:teuthology.orchestra.run.smithi203.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 881 kB/s | 192 kB 00:00 2024-03-29T11:23:04.360 INFO:teuthology.orchestra.run.smithi203.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 12 MB/s | 15 MB 00:01 2024-03-29T11:23:04.993 INFO:teuthology.orchestra.run.smithi203.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 18 MB/s | 34 MB 00:01 2024-03-29T11:23:05.710 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:05.841 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:05.901 INFO:teuthology.orchestra.run.smithi203.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 11 MB/s | 28 MB 00:02 2024-03-29T11:23:05.905 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:23:05.905 INFO:teuthology.orchestra.run.smithi203.stdout:Total 28 MB/s | 76 MB 00:02 2024-03-29T11:23:05.905 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:23:06.321 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:23:06.322 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:23:07.173 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:23:07.174 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:23:09.482 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:23:10.083 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:10.188 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:10.541 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:10.590 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:12.950 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:13.085 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:15.344 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:17.856 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:17.967 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:22.722 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:44.759 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:44.759 INFO:teuthology.orchestra.run.smithi145.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:23:44.759 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:23:44.803 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:45.210 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:45.210 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:45.210 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:45.210 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:45.651 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:45.651 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:23:45.651 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:23:45.651 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.651 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.651 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.651 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:45.651 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:23:45.652 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:23:46.009 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:46.109 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:46.109 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:46.109 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:23:46.110 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:23:46.110 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:23:46.110 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:23:46.110 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:23:46.110 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:46.192 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:46.192 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:46.193 DEBUG:teuthology.orchestra.run.smithi145:> sudo rpm -qi grub2-tools 2024-03-29T11:23:46.230 INFO:teuthology.orchestra.run.smithi145.stdout:Name : grub2-tools 2024-03-29T11:23:46.230 INFO:teuthology.orchestra.run.smithi145.stdout:Epoch : 1 2024-03-29T11:23:46.230 INFO:teuthology.orchestra.run.smithi145.stdout:Version : 2.06 2024-03-29T11:23:46.230 INFO:teuthology.orchestra.run.smithi145.stdout:Release : 68.el9 2024-03-29T11:23:46.230 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: x86_64 2024-03-29T11:23:46.230 INFO:teuthology.orchestra.run.smithi145.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Group : Unspecified 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Size : 8264254 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:License : GPLv3+ 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Packager : builder@centos.org 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Vendor : CentOS 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Summary : Support tools for GRUB. 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:Description : 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:23:46.231 INFO:teuthology.orchestra.run.smithi145.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:23:46.232 INFO:teuthology.orchestra.run.smithi145.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:23:46.232 INFO:teuthology.orchestra.run.smithi145.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:23:46.232 INFO:teuthology.orchestra.run.smithi145.stdout:hardware devices. 2024-03-29T11:23:46.232 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:23:46.232 INFO:teuthology.orchestra.run.smithi145.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:23:46.233 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:23:46.233 INFO:teuthology.task.kernel:Updating grub on smithi145 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:23:46.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:23:46.910 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2024-03-29T11:23:48.315 INFO:teuthology.orchestra.run.smithi145.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:23:48.354 INFO:teuthology.orchestra.run.smithi145.stderr:done 2024-03-29T11:23:48.356 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-03-29T11:23:48.372 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.cIeUuL5vd3 2024-03-29T11:23:48.372 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.cIeUuL5vd3 2024-03-29T11:23:48.440 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.cIeUuL5vd3 2024-03-29T11:23:48.571 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.cIeUuL5vd3 is 6KB 2024-03-29T11:23:48.619 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.cIeUuL5vd3 2024-03-29T11:23:48.633 DEBUG:teuthology.orchestra.run.smithi145:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:23:48.698 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:23:48.698 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:23:48.699 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:23:48.699 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:23:48.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:23:48.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2024-03-29T11:23:52.182 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:52.182 INFO:teuthology.orchestra.run.smithi203.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:23:52.182 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:23:52.226 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:52.611 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:52.611 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:23:52.611 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:23:52.611 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:23:53.041 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:23:53.041 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:23:53.041 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:23:53.041 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:53.041 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:53.041 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:23:53.042 INFO:teuthology.orchestra.run.smithi203.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:23:53.042 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:23:53.042 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:23:53.366 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:53.462 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:53.462 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:53.462 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:23:53.462 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:23:53.462 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:23:53.462 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:23:53.462 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi203.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:23:53.462 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:23:53.542 INFO:teuthology.orchestra.run.smithi203.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:23:53.543 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:23:53.543 DEBUG:teuthology.orchestra.run.smithi203:> sudo rpm -qi grub2-tools 2024-03-29T11:23:53.587 INFO:teuthology.orchestra.run.smithi203.stdout:Name : grub2-tools 2024-03-29T11:23:53.588 INFO:teuthology.orchestra.run.smithi203.stdout:Epoch : 1 2024-03-29T11:23:53.588 INFO:teuthology.orchestra.run.smithi203.stdout:Version : 2.06 2024-03-29T11:23:53.588 INFO:teuthology.orchestra.run.smithi203.stdout:Release : 68.el9 2024-03-29T11:23:53.588 INFO:teuthology.orchestra.run.smithi203.stdout:Architecture: x86_64 2024-03-29T11:23:53.588 INFO:teuthology.orchestra.run.smithi203.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:23:53.588 INFO:teuthology.orchestra.run.smithi203.stdout:Group : Unspecified 2024-03-29T11:23:53.588 INFO:teuthology.orchestra.run.smithi203.stdout:Size : 8264254 2024-03-29T11:23:53.588 INFO:teuthology.orchestra.run.smithi203.stdout:License : GPLv3+ 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:Packager : builder@centos.org 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:Vendor : CentOS 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:Summary : Support tools for GRUB. 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:Description : 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout:hardware devices. 2024-03-29T11:23:53.589 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:23:53.590 INFO:teuthology.orchestra.run.smithi203.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:23:53.591 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:23:53.591 INFO:teuthology.task.kernel:Updating grub on smithi203 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:23:53.591 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:23:54.151 INFO:teuthology.orchestra.run.smithi203.stderr:Generating grub configuration file ... 2024-03-29T11:23:55.533 INFO:teuthology.orchestra.run.smithi203.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:23:55.579 INFO:teuthology.orchestra.run.smithi203.stderr:done 2024-03-29T11:23:55.647 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-29T11:23:55.666 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.dEZwrVQivz 2024-03-29T11:23:55.667 DEBUG:teuthology.orchestra.run.smithi203:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dEZwrVQivz 2024-03-29T11:23:55.740 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.dEZwrVQivz 2024-03-29T11:23:55.876 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.dEZwrVQivz is 6KB 2024-03-29T11:23:55.925 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.dEZwrVQivz 2024-03-29T11:23:55.940 DEBUG:teuthology.orchestra.run.smithi203:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:23:56.004 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:23:56.005 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:23:56.005 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:23:56.005 INFO:teuthology.orchestra.run.smithi203.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:23:56.006 DEBUG:teuthology.orchestra.run.smithi203:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:23:56.156 DEBUG:teuthology.orchestra.run.smithi203:> sudo shutdown -r now 2024-03-29T11:24:18.869 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:24:18.870 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-29T11:24:18.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:26.166 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:24:26.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T11:24:26.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:37.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-29T11:24:44.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-29T11:24:46.281 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-29T11:24:46.282 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:49.342 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-29T11:24:53.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T11:24:53.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:24:56.770 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.203 2024-03-29T11:25:01.347 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-29T11:25:01.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:08.775 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T11:25:08.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:19.775 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-03-29T11:25:34.786 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-29T11:25:34.787 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:25:35.215 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-03-29T11:25:35.767 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-29T11:25:35.768 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:25:35.768 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-03-29T11:25:35.788 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:25:35.789 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:25:35.789 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:25:35.789 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:25:36.789 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:25:36.790 DEBUG:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:25:36.904 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2024-03-29T11:25:36.935 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:08.778 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:26:23.779 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T11:26:23.781 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:26:24.073 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-29T11:26:24.452 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T11:26:24.452 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:26:24.453 DEBUG:teuthology.orchestra.run.smithi203:> uname -r 2024-03-29T11:26:24.468 INFO:teuthology.orchestra.run.smithi203.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:26:24.468 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:26:24.468 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:26:24.468 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:26:25.469 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:26:25.469 DEBUG:teuthology.orchestra.run.smithi203:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:26:25.536 INFO:teuthology.orchestra.run.smithi203.stdout:ttyS1 2024-03-29T11:26:25.561 DEBUG:teuthology.parallel:result is None 2024-03-29T11:26:25.562 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:26:25.569 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:26:25.569 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:25.572 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:26:25.589 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:26:25.610 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:26:25.715 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:26:25.715 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:25.717 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:26:25.747 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:26:25.777 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:26:25.777 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:25.784 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:26:25.825 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:25.831 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:25.839 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:25.845 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:26:25.847 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:26:25.854 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:26:25.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:25.856 DEBUG:teuthology.orchestra.run.smithi203:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:26:25.911 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:26:25.920 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:26:25.921 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:25.926 DEBUG:teuthology.orchestra.run.smithi203:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:26:25.967 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:26.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:26.074 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:26.152 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:26.198 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:26:26.198 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:26.265 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:26.308 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:26:26.364 DEBUG:teuthology.orchestra.run.smithi203:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:26.449 DEBUG:teuthology.orchestra.run.smithi203:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:26:26.492 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:26:26.492 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:26:26.560 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2024-03-29T11:26:26.563 DEBUG:teuthology.orchestra.run.smithi203:> sudo service rsyslog restart 2024-03-29T11:26:26.626 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:26.656 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:26:26.957 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:26:26.965 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:26:26.965 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:26:27.011 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:26:27.043 DEBUG:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2024-03-29T11:26:27.089 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: 2024-03-29T11:26:27.090 DEBUG:teuthology.orchestra.run.smithi203:> sudo service auditd rotate 2024-03-29T11:26:27.154 INFO:teuthology.orchestra.run.smithi203.stdout:Rotating logs: 2024-03-29T11:26:27.155 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:26:27.156 DEBUG:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2024-03-29T11:26:27.182 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2024-03-29T11:26:27.183 DEBUG:teuthology.orchestra.run.smithi203:> /usr/sbin/getenforce 2024-03-29T11:26:27.202 INFO:teuthology.orchestra.run.smithi203.stdout:Permissive 2024-03-29T11:26:27.202 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi145.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi203.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:26:27.203 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:27.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:27.251 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:26:27.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:26:27.279 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:26:27.279 DEBUG:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:27.335 DEBUG:teuthology.orchestra.run.smithi203:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:26:27.370 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:26:27.380 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:26:27.399 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:26:27.400 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi145.front.sepia.ceph.com,smithi203.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:30:52.968 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi145.front.sepia.ceph.com'), Remote(name='ubuntu@smithi203.front.sepia.ceph.com')] 2024-03-29T11:30:52.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-29T11:30:52.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:53.058 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-03-29T11:30:53.094 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-03-29T11:30:53.094 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T11:30:53.095 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi203.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:30:53.182 DEBUG:teuthology.orchestra.run.smithi203:> true 2024-03-29T11:30:53.257 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi203.front.sepia.ceph.com' 2024-03-29T11:30:53.257 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:30:53.267 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:30:53.267 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:30:53.268 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:30:53.270 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:30:53.270 DEBUG:teuthology.orchestra.run.smithi203:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:30:53.298 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:30:53.315 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:30:53.327 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:30:53.344 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: ntpd: command not found 2024-03-29T11:30:53.344 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:30:53.358 INFO:teuthology.orchestra.run.smithi145.stdout:506 Cannot talk to daemon 2024-03-29T11:30:53.373 INFO:teuthology.orchestra.run.smithi203.stderr:sudo: ntpd: command not found 2024-03-29T11:30:53.374 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:30:53.387 INFO:teuthology.orchestra.run.smithi203.stdout:506 Cannot talk to daemon 2024-03-29T11:30:53.390 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:30:53.403 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:30:53.420 INFO:teuthology.orchestra.run.smithi203.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:30:53.453 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:30:53.458 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:30:53.458 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-03-29T11:30:53.458 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:53.458 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:53.458 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:53.458 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:53.474 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:30:53.477 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:30:53.477 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-29T11:30:53.477 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:53.477 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:53.477 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:53.478 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:30:53.478 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:30:53.486 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:30:53.487 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:30:53.487 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:30:53.487 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:30:53.493 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:30:53.493 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:30:53.494 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:30:53.494 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:30:53.496 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:30:53.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:30:53.638 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:30:53.638 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:30:53.747 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:30:53.747 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:30:53.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:30:53.760 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:30:53.760 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:30:53.775 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:30:53.775 DEBUG:teuthology.orchestra.run.smithi145:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:30:53.843 DEBUG:teuthology.orchestra.run.smithi145:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:30:53.923 DEBUG:teuthology.orchestra.run.smithi145:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:30:53.930 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:30:53.930 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:30:53.931 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:30:53.960 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:30:53.961 DEBUG:teuthology.orchestra.run.smithi203:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:30:53.990 INFO:teuthology.orchestra.run.smithi145.stdout:check_obsoletes = 1 2024-03-29T11:30:53.991 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum clean all 2024-03-29T11:30:54.030 DEBUG:teuthology.orchestra.run.smithi203:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:30:54.111 DEBUG:teuthology.orchestra.run.smithi203:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:30:54.178 INFO:teuthology.orchestra.run.smithi203.stdout:check_obsoletes = 1 2024-03-29T11:30:54.179 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum clean all 2024-03-29T11:30:54.246 INFO:teuthology.orchestra.run.smithi145.stdout:74 files removed 2024-03-29T11:30:54.273 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-radosgw 2024-03-29T11:30:54.424 INFO:teuthology.orchestra.run.smithi203.stdout:74 files removed 2024-03-29T11:30:54.451 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-radosgw 2024-03-29T11:30:55.039 INFO:teuthology.orchestra.run.smithi145.stdout:ceph packages for x86_64 176 kB/s | 82 kB 00:00 2024-03-29T11:30:55.098 INFO:teuthology.orchestra.run.smithi203.stdout:ceph packages for x86_64 237 kB/s | 82 kB 00:00 2024-03-29T11:30:55.431 INFO:teuthology.orchestra.run.smithi145.stdout:ceph noarch packages 29 kB/s | 10 kB 00:00 2024-03-29T11:30:55.624 INFO:teuthology.orchestra.run.smithi203.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-29T11:30:55.849 INFO:teuthology.orchestra.run.smithi145.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-29T11:30:56.007 INFO:teuthology.orchestra.run.smithi203.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-29T11:30:56.301 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-29T11:30:56.443 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-29T11:30:57.718 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-29T11:30:57.852 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-03-29T11:30:59.984 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-03-29T11:31:00.242 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 3.7 MB/s | 8.0 MB 00:02 2024-03-29T11:31:02.002 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-29T11:31:06.330 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 6.1 MB 00:00 2024-03-29T11:31:07.729 INFO:teuthology.orchestra.run.smithi203.stdout:CentOS Stream 9 - Extras packages 61 kB/s | 16 kB 00:00 2024-03-29T11:31:09.347 INFO:teuthology.orchestra.run.smithi203.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-29T11:31:14.374 INFO:teuthology.orchestra.run.smithi203.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-29T11:31:16.996 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrading: 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:17.000 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:17.001 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout:Installing weak dependencies: 2024-03-29T11:31:17.002 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:31:17.003 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:31:17.003 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:31:17.003 INFO:teuthology.orchestra.run.smithi203.stdout:Install 30 Packages 2024-03-29T11:31:17.003 INFO:teuthology.orchestra.run.smithi203.stdout:Upgrade 2 Packages 2024-03-29T11:31:17.003 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:31:17.003 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 58 M 2024-03-29T11:31:17.003 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:31:22.523 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 913 kB/s | 19 MB 00:21 2024-03-29T11:31:24.689 INFO:teuthology.orchestra.run.smithi203.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 2.7 MB/s | 20 MB 00:07 2024-03-29T11:31:24.806 INFO:teuthology.orchestra.run.smithi203.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-29T11:31:25.598 INFO:teuthology.orchestra.run.smithi203.stdout:(3/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 654 kB/s | 5.3 MB 00:08 2024-03-29T11:31:25.698 INFO:teuthology.orchestra.run.smithi203.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 832 kB/s | 742 kB 00:00 2024-03-29T11:31:26.065 INFO:teuthology.orchestra.run.smithi203.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 1.0 MB/s | 498 kB 00:00 2024-03-29T11:31:26.157 INFO:teuthology.orchestra.run.smithi203.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 493 kB/s | 45 kB 00:00 2024-03-29T11:31:26.333 INFO:teuthology.orchestra.run.smithi203.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 751 kB/s | 131 kB 00:00 2024-03-29T11:31:26.466 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.1 MB 00:00 2024-03-29T11:31:26.533 INFO:teuthology.orchestra.run.smithi203.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 811 kB/s | 162 kB 00:00 2024-03-29T11:31:26.849 INFO:teuthology.orchestra.run.smithi203.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 1.0 MB/s | 320 kB 00:00 2024-03-29T11:31:27.141 INFO:teuthology.orchestra.run.smithi203.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 1.0 MB/s | 298 kB 00:00 2024-03-29T11:31:27.292 INFO:teuthology.orchestra.run.smithi203.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 662 kB/s | 99 kB 00:00 2024-03-29T11:31:27.409 INFO:teuthology.orchestra.run.smithi203.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 910 kB/s | 106 kB 00:00 2024-03-29T11:31:27.451 INFO:teuthology.orchestra.run.smithi203.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T11:31:27.484 INFO:teuthology.orchestra.run.smithi203.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-29T11:31:27.543 INFO:teuthology.orchestra.run.smithi203.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T11:31:27.585 INFO:teuthology.orchestra.run.smithi203.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:31:27.618 INFO:teuthology.orchestra.run.smithi203.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:31:27.661 INFO:teuthology.orchestra.run.smithi203.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T11:31:27.694 INFO:teuthology.orchestra.run.smithi203.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T11:31:27.728 INFO:teuthology.orchestra.run.smithi203.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T11:31:27.928 INFO:teuthology.orchestra.run.smithi203.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-03-29T11:31:28.104 INFO:teuthology.orchestra.run.smithi203.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 25 MB/s | 4.4 MB 00:00 2024-03-29T11:31:28.154 INFO:teuthology.orchestra.run.smithi203.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 523 kB/s | 26 kB 00:00 2024-03-29T11:31:28.196 INFO:teuthology.orchestra.run.smithi203.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-29T11:31:28.246 INFO:teuthology.orchestra.run.smithi203.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-29T11:31:28.300 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-29T11:31:28.304 INFO:teuthology.orchestra.run.smithi203.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 14 MB/s | 838 kB 00:00 2024-03-29T11:31:28.430 INFO:teuthology.orchestra.run.smithi203.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-29T11:31:28.514 INFO:teuthology.orchestra.run.smithi203.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-29T11:31:29.152 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 29 MB/s | 21 MB 00:00 2024-03-29T11:31:30.142 INFO:teuthology.orchestra.run.smithi203.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 5.1 MB 00:04 2024-03-29T11:31:30.814 INFO:teuthology.orchestra.run.smithi203.stdout:(30/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 734 kB/s | 9.7 MB 00:13 2024-03-29T11:31:31.239 INFO:teuthology.orchestra.run.smithi203.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.2 MB/s | 3.4 MB 00:02 2024-03-29T11:31:34.038 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-29T11:31:34.789 INFO:teuthology.orchestra.run.smithi203.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 689 kB/s | 3.1 MB 00:04 2024-03-29T11:31:34.794 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:34.794 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.3 MB/s | 58 MB 00:17 2024-03-29T11:31:34.794 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:31:34.833 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:31:34.833 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:31:35.345 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:31:35.345 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:31:36.722 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:31:36.725 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:31:36.725 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:31:36.725 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:31:36.725 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:31:36.725 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:31:36.725 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrading: 2024-03-29T11:31:36.725 INFO:teuthology.orchestra.run.smithi145.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: boost-program-options 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:31:36.726 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:31:36.727 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout:Installing weak dependencies: 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:31:36.728 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:31:36.729 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:31:36.729 INFO:teuthology.orchestra.run.smithi145.stdout:Install 30 Packages 2024-03-29T11:31:36.729 INFO:teuthology.orchestra.run.smithi145.stdout:Upgrade 2 Packages 2024-03-29T11:31:36.729 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:31:36.729 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 58 M 2024-03-29T11:31:36.729 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:31:36.876 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:31:37.011 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:37.044 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:37.074 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:37.109 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:37.125 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:37.276 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:37.331 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:37.679 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:37.738 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:37.830 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:37.920 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:37.956 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:38.018 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:38.048 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:38.178 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:38.350 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:38.413 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:38.448 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:38.559 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:38.625 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:38.665 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:38.689 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:38.718 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:38.759 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:39.076 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:39.106 INFO:teuthology.orchestra.run.smithi203.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:39.180 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:39.211 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:39.253 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:39.717 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:39.742 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:39.813 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:39.881 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:39.936 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:40.986 INFO:teuthology.orchestra.run.smithi145.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.3 MB/s | 5.3 MB 00:04 2024-03-29T11:31:41.415 INFO:teuthology.orchestra.run.smithi145.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-29T11:31:41.415 INFO:teuthology.orchestra.run.smithi145.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 2.5 MB/s | 742 kB 00:00 2024-03-29T11:31:41.595 INFO:teuthology.orchestra.run.smithi145.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 2.4 MB/s | 498 kB 00:00 2024-03-29T11:31:42.147 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:42.191 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:42.587 INFO:teuthology.orchestra.run.smithi145.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 9.7 MB 00:05 2024-03-29T11:31:42.687 INFO:teuthology.orchestra.run.smithi145.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 452 kB/s | 45 kB 00:00 2024-03-29T11:31:42.762 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:42.796 INFO:teuthology.orchestra.run.smithi145.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.2 MB/s | 131 kB 00:00 2024-03-29T11:31:42.803 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:42.892 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:42.892 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:42.893 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:42.893 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:31:42.912 INFO:teuthology.orchestra.run.smithi145.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.4 MB/s | 162 kB 00:00 2024-03-29T11:31:42.929 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:43.071 INFO:teuthology.orchestra.run.smithi145.stdout:(9/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 3.5 MB/s | 5.1 MB 00:01 2024-03-29T11:31:43.139 INFO:teuthology.orchestra.run.smithi145.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 1.4 MB/s | 320 kB 00:00 2024-03-29T11:31:43.188 INFO:teuthology.orchestra.run.smithi145.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.5 MB/s | 298 kB 00:00 2024-03-29T11:31:43.247 INFO:teuthology.orchestra.run.smithi145.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 913 kB/s | 99 kB 00:00 2024-03-29T11:31:43.306 INFO:teuthology.orchestra.run.smithi145.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 905 kB/s | 106 kB 00:00 2024-03-29T11:31:43.331 INFO:teuthology.orchestra.run.smithi145.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-29T11:31:43.356 INFO:teuthology.orchestra.run.smithi145.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 884 kB/s | 44 kB 00:00 2024-03-29T11:31:43.390 INFO:teuthology.orchestra.run.smithi145.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T11:31:43.415 INFO:teuthology.orchestra.run.smithi145.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-29T11:31:43.440 INFO:teuthology.orchestra.run.smithi145.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 826 kB/s | 41 kB 00:00 2024-03-29T11:31:43.466 INFO:teuthology.orchestra.run.smithi145.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:31:43.491 INFO:teuthology.orchestra.run.smithi145.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 700 kB/s | 35 kB 00:00 2024-03-29T11:31:43.516 INFO:teuthology.orchestra.run.smithi145.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 924 kB/s | 46 kB 00:00 2024-03-29T11:31:43.758 INFO:teuthology.orchestra.run.smithi145.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-29T11:31:43.833 INFO:teuthology.orchestra.run.smithi145.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 352 kB/s | 26 kB 00:00 2024-03-29T11:31:43.909 INFO:teuthology.orchestra.run.smithi145.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 649 kB/s | 49 kB 00:00 2024-03-29T11:31:43.992 INFO:teuthology.orchestra.run.smithi145.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 9.3 MB/s | 4.4 MB 00:00 2024-03-29T11:31:44.018 INFO:teuthology.orchestra.run.smithi145.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 623 kB/s | 67 kB 00:00 2024-03-29T11:31:44.093 INFO:teuthology.orchestra.run.smithi145.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-29T11:31:44.118 INFO:teuthology.orchestra.run.smithi145.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-29T11:31:44.226 INFO:teuthology.orchestra.run.smithi145.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 12 MB/s | 1.6 MB 00:00 2024-03-29T11:31:45.402 INFO:teuthology.orchestra.run.smithi145.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 2.6 MB/s | 3.4 MB 00:01 2024-03-29T11:31:46.177 INFO:teuthology.orchestra.run.smithi145.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 3.1 MB 00:01 2024-03-29T11:31:51.176 INFO:teuthology.orchestra.run.smithi145.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 20 MB 00:14 2024-03-29T11:31:51.182 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:31:51.182 INFO:teuthology.orchestra.run.smithi145.stdout:Total 4.0 MB/s | 58 MB 00:14 2024-03-29T11:31:51.182 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:31:51.219 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:31:51.219 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:31:51.691 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:31:51.691 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:31:52.300 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:52.301 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /sys 2024-03-29T11:31:52.301 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /proc 2024-03-29T11:31:52.301 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /mnt 2024-03-29T11:31:52.301 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /var/tmp 2024-03-29T11:31:52.301 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /home 2024-03-29T11:31:52.301 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /root 2024-03-29T11:31:52.301 INFO:teuthology.orchestra.run.smithi203.stdout:skipping the directory /tmp 2024-03-29T11:31:52.301 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:31:53.142 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:31:53.250 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:53.254 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:31:53.360 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:31:53.361 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:31:53.361 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:31:53.361 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:53.361 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:31:53.362 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:31:53.362 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:53.362 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:31:53.362 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:31:53.362 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:53.364 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:31:53.371 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:31:53.371 INFO:teuthology.orchestra.run.smithi203.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:53.547 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:53.605 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:31:53.939 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:31:53.992 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:54.075 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:54.162 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:31:54.185 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:54.248 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:31:54.277 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:31:54.321 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:31:54.389 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:54.413 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:54.468 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:31:54.503 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:54.521 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:31:54.522 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:31:54.523 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:31:54.524 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:31:54.525 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:31:54.611 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:31:54.663 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:31:54.705 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:31:54.739 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:31:54.765 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:31:54.797 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:31:55.123 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:31:55.161 INFO:teuthology.orchestra.run.smithi145.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:55.250 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:31:55.282 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:31:55.323 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:31:55.671 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:31:55.671 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:31:55.671 INFO:teuthology.orchestra.run.smithi203.stdout:Upgraded: 2024-03-29T11:31:55.671 INFO:teuthology.orchestra.run.smithi203.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.671 INFO:teuthology.orchestra.run.smithi203.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:31:55.672 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.673 INFO:teuthology.orchestra.run.smithi203.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:31:55.674 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:31:55.776 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:31:55.797 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:55.851 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:31:55.922 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:31:55.957 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:31:55.963 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-test 2024-03-29T11:31:56.521 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:31:56.679 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:31:56.680 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:31:56.681 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:31:56.681 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:31:56.681 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-03-29T11:31:56.681 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:31:56.681 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 48 M 2024-03-29T11:31:56.681 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 204 M 2024-03-29T11:31:56.681 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:31:56.866 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-29T11:31:56.916 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-29T11:31:56.950 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-29T11:31:57.159 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 898 kB/s | 187 kB 00:00 2024-03-29T11:31:57.370 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 163 kB 00:00 2024-03-29T11:31:58.131 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:58.173 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:58.753 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:31:58.791 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:58.883 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:31:58.883 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:31:58.883 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:31:58.883 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:31:58.917 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:06.934 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 4.6 MB/s | 47 MB 00:10 2024-03-29T11:32:06.936 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:06.937 INFO:teuthology.orchestra.run.smithi203.stdout:Total 4.7 MB/s | 48 MB 00:10 2024-03-29T11:32:06.950 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:32:06.968 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:32:06.968 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:32:07.359 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:32:07.360 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:32:07.932 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:32:08.017 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:08.057 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:08.099 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:08.151 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:08.157 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:08.157 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /sys 2024-03-29T11:32:08.157 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /proc 2024-03-29T11:32:08.157 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /mnt 2024-03-29T11:32:08.157 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /var/tmp 2024-03-29T11:32:08.157 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /home 2024-03-29T11:32:08.157 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /root 2024-03-29T11:32:08.158 INFO:teuthology.orchestra.run.smithi145.stdout:skipping the directory /tmp 2024-03-29T11:32:08.158 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:08.187 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:09.060 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:09.084 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:32:09.084 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:09.084 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:32:09.084 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:09.084 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:32:09.084 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:09.085 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:09.172 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:32:09.172 INFO:teuthology.orchestra.run.smithi145.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:10.310 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:32:10.311 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:32:10.315 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:32:10.316 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:32:11.214 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:32:11.214 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:11.214 INFO:teuthology.orchestra.run.smithi145.stdout:Upgraded: 2024-03-29T11:32:11.214 INFO:teuthology.orchestra.run.smithi145.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:32:11.215 INFO:teuthology.orchestra.run.smithi145.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.216 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:11.217 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:32:11.217 INFO:teuthology.orchestra.run.smithi145.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:32:11.217 INFO:teuthology.orchestra.run.smithi145.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:32:11.217 INFO:teuthology.orchestra.run.smithi145.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:32:11.217 INFO:teuthology.orchestra.run.smithi145.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:32:11.217 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:11.217 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:32:11.557 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-test 2024-03-29T11:32:12.118 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:00:39 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:32:12.279 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:32:12.280 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi145.stdout:Install 6 Packages 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 48 M 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 204 M 2024-03-29T11:32:12.281 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:32:12.553 INFO:teuthology.orchestra.run.smithi145.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:32:12.603 INFO:teuthology.orchestra.run.smithi145.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 300 kB 00:00 2024-03-29T11:32:12.647 INFO:teuthology.orchestra.run.smithi145.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T11:32:12.667 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:12.730 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:12.852 INFO:teuthology.orchestra.run.smithi145.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.0 MB/s | 187 kB 00:00 2024-03-29T11:32:12.937 INFO:teuthology.orchestra.run.smithi145.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 301 kB/s | 163 kB 00:00 2024-03-29T11:32:15.184 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:15.185 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:15.185 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:15.185 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:15.185 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:15.185 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:15.550 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:15.550 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:15.550 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:32:15.550 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:15.550 INFO:teuthology.orchestra.run.smithi203.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:15.550 INFO:teuthology.orchestra.run.smithi203.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:15.550 INFO:teuthology.orchestra.run.smithi203.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:15.550 INFO:teuthology.orchestra.run.smithi203.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:15.551 INFO:teuthology.orchestra.run.smithi203.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:15.551 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:15.551 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:32:15.823 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph 2024-03-29T11:32:16.389 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:32:16.542 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:32:16.545 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-03-29T11:32:16.545 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:32:16.545 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-03-29T11:32:16.545 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:32:16.545 INFO:teuthology.orchestra.run.smithi203.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:16.545 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:16.546 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:16.547 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout:========================================================================================= 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout:Install 36 Packages 2024-03-29T11:32:16.548 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:16.549 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 30 M 2024-03-29T11:32:16.549 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 107 M 2024-03-29T11:32:16.549 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:32:17.282 INFO:teuthology.orchestra.run.smithi203.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 22 kB/s | 6.6 kB 00:00 2024-03-29T11:32:19.949 INFO:teuthology.orchestra.run.smithi203.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 513 kB/s | 1.5 MB 00:02 2024-03-29T11:32:19.999 INFO:teuthology.orchestra.run.smithi203.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 720 kB/s | 2.1 MB 00:03 2024-03-29T11:32:20.488 INFO:teuthology.orchestra.run.smithi203.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 512 kB/s | 247 kB 00:00 2024-03-29T11:32:20.608 INFO:teuthology.orchestra.run.smithi203.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-29T11:32:20.650 INFO:teuthology.orchestra.run.smithi203.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T11:32:20.684 INFO:teuthology.orchestra.run.smithi203.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 949 kB/s | 32 kB 00:00 2024-03-29T11:32:20.717 INFO:teuthology.orchestra.run.smithi203.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T11:32:20.751 INFO:teuthology.orchestra.run.smithi203.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-29T11:32:20.793 INFO:teuthology.orchestra.run.smithi203.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-29T11:32:20.860 INFO:teuthology.orchestra.run.smithi203.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-29T11:32:20.895 INFO:teuthology.orchestra.run.smithi203.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:32:20.927 INFO:teuthology.orchestra.run.smithi203.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-29T11:32:20.962 INFO:teuthology.orchestra.run.smithi203.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:32:21.095 INFO:teuthology.orchestra.run.smithi203.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 937 kB/s | 125 kB 00:00 2024-03-29T11:32:21.363 INFO:teuthology.orchestra.run.smithi203.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 84 kB/s | 22 kB 00:00 2024-03-29T11:32:21.595 INFO:teuthology.orchestra.run.smithi203.stdout:(17/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 4.7 MB 00:04 2024-03-29T11:32:21.621 INFO:teuthology.orchestra.run.smithi203.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 584 kB/s | 151 kB 00:00 2024-03-29T11:32:21.721 INFO:teuthology.orchestra.run.smithi203.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-29T11:32:21.738 INFO:teuthology.orchestra.run.smithi203.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 305 kB/s | 43 kB 00:00 2024-03-29T11:32:21.788 INFO:teuthology.orchestra.run.smithi203.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-29T11:32:21.815 INFO:teuthology.orchestra.run.smithi203.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-29T11:32:21.848 INFO:teuthology.orchestra.run.smithi203.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 299 kB/s | 18 kB 00:00 2024-03-29T11:32:21.865 INFO:teuthology.orchestra.run.smithi203.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 459 kB/s | 23 kB 00:00 2024-03-29T11:32:21.899 INFO:teuthology.orchestra.run.smithi203.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-29T11:32:21.915 INFO:teuthology.orchestra.run.smithi203.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-03-29T11:32:21.974 INFO:teuthology.orchestra.run.smithi203.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-29T11:32:22.008 INFO:teuthology.orchestra.run.smithi203.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 425 kB/s | 46 kB 00:00 2024-03-29T11:32:22.042 INFO:teuthology.orchestra.run.smithi203.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T11:32:22.068 INFO:teuthology.orchestra.run.smithi203.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 280 kB/s | 16 kB 00:00 2024-03-29T11:32:22.101 INFO:teuthology.orchestra.run.smithi203.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-29T11:32:22.126 INFO:teuthology.orchestra.run.smithi203.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 609 kB/s | 36 kB 00:00 2024-03-29T11:32:22.176 INFO:teuthology.orchestra.run.smithi203.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T11:32:22.226 INFO:teuthology.orchestra.run.smithi203.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-03-29T11:32:22.268 INFO:teuthology.orchestra.run.smithi203.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-29T11:32:36.801 INFO:teuthology.orchestra.run.smithi203.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.0 MB/s | 17 MB 00:16 2024-03-29T11:32:36.804 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:36.804 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.5 MB/s | 30 MB 00:20 2024-03-29T11:32:36.829 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:32:36.872 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:32:36.873 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:32:37.284 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:32:37.284 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:32:38.643 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:32:38.703 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:32:38.736 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:32:38.763 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:32:38.836 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:32:38.887 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:32:38.929 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:32:39.020 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:32:39.107 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:32:39.148 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:32:39.220 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:32:39.274 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:32:39.321 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:32:39.379 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:32:39.434 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:32:39.627 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:32:39.676 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:32:39.707 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:32:39.741 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:32:39.771 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:32:39.848 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:32:39.882 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:39.915 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:32:39.964 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:32:40.007 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:32:40.042 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:32:40.131 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:32:40.165 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:32:40.237 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:32:40.382 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:32:40.508 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:40.666 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:32:40.696 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:40.721 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:32:40.721 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:40.722 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:32:40.722 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:40.722 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:32:40.722 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:42.147 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:42.173 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:32:42.173 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:42.174 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:32:42.174 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:42.174 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:32:42.174 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:42.624 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:42.649 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:32:42.649 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:42.649 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:32:42.649 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:42.649 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:32:42.649 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:42.891 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:42.913 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:32:42.913 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:42.913 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:32:42.913 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:42.914 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:32:42.914 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:42.990 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:44.787 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:32:44.788 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:32:44.789 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:32:44.791 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:32:45.862 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:32:45.862 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:45.862 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:32:45.862 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.862 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.862 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:32:45.863 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:32:45.864 INFO:teuthology.orchestra.run.smithi203.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:32:45.865 INFO:teuthology.orchestra.run.smithi203.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:32:45.865 INFO:teuthology.orchestra.run.smithi203.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:32:45.865 INFO:teuthology.orchestra.run.smithi203.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:32:45.865 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:45.865 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:32:46.100 INFO:teuthology.orchestra.run.smithi145.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 47 MB 00:33 2024-03-29T11:32:46.102 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:46.103 INFO:teuthology.orchestra.run.smithi145.stdout:Total 1.4 MB/s | 48 MB 00:33 2024-03-29T11:32:46.116 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:32:46.137 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:32:46.137 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:32:46.223 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-base 2024-03-29T11:32:46.527 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:32:46.528 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:32:46.802 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:32:46.900 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:46.952 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:32:46.953 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-29T11:32:46.953 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:32:47.032 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install cephadm 2024-03-29T11:32:47.115 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:32:47.183 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:32:47.219 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:32:47.252 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:47.298 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:32:47.328 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:47.595 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:32:47.744 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:32:47.744 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 747 k 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 754 k 2024-03-29T11:32:47.745 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:32:48.210 INFO:teuthology.orchestra.run.smithi203.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.6 MB/s | 747 kB 00:00 2024-03-29T11:32:48.210 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:48.211 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.6 MB/s | 747 kB 00:00 2024-03-29T11:32:48.211 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:32:48.215 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:32:48.215 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:32:48.222 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:32:48.222 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:32:48.307 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:32:48.701 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:48.789 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:49.157 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:49.452 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:32:49.452 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:49.452 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:32:49.452 INFO:teuthology.orchestra.run.smithi203.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:49.452 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:49.452 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:32:49.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:32:50.248 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:32:50.398 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-29T11:32:50.399 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:50.400 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 145 k 2024-03-29T11:32:50.400 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 436 k 2024-03-29T11:32:50.400 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:32:50.760 INFO:teuthology.orchestra.run.smithi203.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 403 kB/s | 145 kB 00:00 2024-03-29T11:32:50.760 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:50.761 INFO:teuthology.orchestra.run.smithi203.stdout:Total 402 kB/s | 145 kB 00:00 2024-03-29T11:32:50.761 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:32:50.769 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:32:50.769 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:32:50.821 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:32:50.822 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:32:50.963 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:32:51.036 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:51.060 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:51.061 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:32:51.061 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:32:51.061 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:51.775 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:32:51.852 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:32:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:52.029 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:32:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:52.029 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:52.029 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:32:52.289 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr 2024-03-29T11:32:52.860 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:32:52.963 INFO:teuthology.orchestra.run.smithi203.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:32:53.014 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:32:53.015 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-29T11:32:53.015 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:32:53.089 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:32:53.655 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:32:53.817 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:32:53.818 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-03-29T11:32:53.818 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:32:53.818 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-03-29T11:32:53.818 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:32:53.818 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:32:53.819 INFO:teuthology.orchestra.run.smithi203.stdout:====================================================================================== 2024-03-29T11:32:53.820 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-03-29T11:32:53.820 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:53.820 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.9 M 2024-03-29T11:32:53.820 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-03-29T11:32:53.820 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:32:54.280 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:32:54.280 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:32:54.280 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:32:54.281 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:32:54.281 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:32:54.281 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:32:54.391 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 119 kB/s | 31 kB 00:00 2024-03-29T11:32:54.407 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 54 kB/s | 15 kB 00:00 2024-03-29T11:32:54.549 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T11:32:54.591 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T11:32:54.707 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:32:54.707 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:54.707 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:32:54.708 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:54.708 INFO:teuthology.orchestra.run.smithi145.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:32:54.708 INFO:teuthology.orchestra.run.smithi145.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:32:54.708 INFO:teuthology.orchestra.run.smithi145.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:32:54.708 INFO:teuthology.orchestra.run.smithi145.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:32:54.708 INFO:teuthology.orchestra.run.smithi145.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:32:54.708 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:54.708 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:32:55.032 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph 2024-03-29T11:32:55.200 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 1.7 MB/s | 1.8 MB 00:01 2024-03-29T11:32:55.350 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 722 kB/s | 548 kB 00:00 2024-03-29T11:32:55.384 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.0 MB/s | 863 kB 00:00 2024-03-29T11:32:55.576 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): grpc-data-1.46.7-10.el9.noarch.rpm 52 kB/s | 19 kB 00:00 2024-03-29T11:32:55.601 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:32:55.710 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 5.7 MB/s | 2.0 MB 00:00 2024-03-29T11:32:55.735 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 195 kB/s | 31 kB 00:00 2024-03-29T11:32:55.765 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout:========================================================================================= 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout:========================================================================================= 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:32:55.768 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:32:55.769 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:32:55.770 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:32:55.771 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:32:55.771 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:32:55.771 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:55.771 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:32:55.771 INFO:teuthology.orchestra.run.smithi145.stdout:========================================================================================= 2024-03-29T11:32:55.771 INFO:teuthology.orchestra.run.smithi145.stdout:Install 36 Packages 2024-03-29T11:32:55.771 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:32:55.771 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 30 M 2024-03-29T11:32:55.772 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 107 M 2024-03-29T11:32:55.772 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:32:55.827 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-29T11:32:55.985 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): python3-grpcio-tools-1.46.7-10.el9.x86 240 kB/s | 144 kB 00:00 2024-03-29T11:32:55.991 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:32:55.991 INFO:teuthology.orchestra.run.smithi203.stdout:Total 3.2 MB/s | 6.9 MB 00:02 2024-03-29T11:32:56.010 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:32:56.028 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:32:56.028 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:32:56.212 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:32:56.212 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:32:56.620 INFO:teuthology.orchestra.run.smithi145.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 19 kB/s | 6.6 kB 00:00 2024-03-29T11:32:56.721 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:32:56.786 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:32:56.838 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:32:56.893 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:32:56.932 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:32:57.033 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:32:57.175 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:32:57.262 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:32:57.313 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:32:57.349 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:32:57.381 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:32:57.635 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:32:57.673 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:32:58.070 INFO:teuthology.orchestra.run.smithi145.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 851 kB/s | 1.5 MB 00:01 2024-03-29T11:32:58.480 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:32:58.480 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:32:58.480 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:32:58.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:32:58.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:32:58.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:32:58.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:32:58.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:32:58.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:32:58.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:32:58.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:32:58.481 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:32:58.853 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:32:58.853 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:58.853 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:32:58.853 INFO:teuthology.orchestra.run.smithi203.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:32:58.853 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:58.853 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:58.853 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:32:58.853 INFO:teuthology.orchestra.run.smithi203.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:32:58.854 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:32:58.854 INFO:teuthology.orchestra.run.smithi203.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:32:58.854 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:32:58.854 INFO:teuthology.orchestra.run.smithi203.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:32:58.854 INFO:teuthology.orchestra.run.smithi203.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:32:58.854 INFO:teuthology.orchestra.run.smithi203.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:32:58.854 INFO:teuthology.orchestra.run.smithi203.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:32:58.854 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:58.854 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:32:58.920 INFO:teuthology.orchestra.run.smithi145.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 822 kB/s | 2.1 MB 00:02 2024-03-29T11:32:59.122 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:32:59.295 INFO:teuthology.orchestra.run.smithi145.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 660 kB/s | 247 kB 00:00 2024-03-29T11:32:59.487 INFO:teuthology.orchestra.run.smithi145.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 976 kB/s | 187 kB 00:00 2024-03-29T11:32:59.537 INFO:teuthology.orchestra.run.smithi145.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-29T11:32:59.571 INFO:teuthology.orchestra.run.smithi145.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-29T11:32:59.630 INFO:teuthology.orchestra.run.smithi145.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 813 kB/s | 47 kB 00:00 2024-03-29T11:32:59.680 INFO:teuthology.orchestra.run.smithi145.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 774 kB/s | 39 kB 00:00 2024-03-29T11:32:59.683 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:32:59.747 INFO:teuthology.orchestra.run.smithi145.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.5 MB/s | 241 kB 00:00 2024-03-29T11:32:59.822 INFO:teuthology.orchestra.run.smithi145.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T11:32:59.838 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:32:59.839 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout:============================================================================================ 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout:Install 12 Packages 2024-03-29T11:32:59.840 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:32:59.841 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 37 M 2024-03-29T11:32:59.841 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 209 M 2024-03-29T11:32:59.841 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:32:59.856 INFO:teuthology.orchestra.run.smithi145.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-29T11:32:59.889 INFO:teuthology.orchestra.run.smithi145.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-29T11:32:59.923 INFO:teuthology.orchestra.run.smithi145.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-29T11:33:00.074 INFO:teuthology.orchestra.run.smithi145.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 832 kB/s | 125 kB 00:00 2024-03-29T11:33:00.128 INFO:teuthology.orchestra.run.smithi203.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 234 kB/s | 32 kB 00:00 2024-03-29T11:33:00.161 INFO:teuthology.orchestra.run.smithi203.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 517 kB/s | 17 kB 00:00 2024-03-29T11:33:00.195 INFO:teuthology.orchestra.run.smithi203.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-29T11:33:00.370 INFO:teuthology.orchestra.run.smithi203.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-29T11:33:00.504 INFO:teuthology.orchestra.run.smithi203.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-29T11:33:00.571 INFO:teuthology.orchestra.run.smithi203.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-29T11:33:00.596 INFO:teuthology.orchestra.run.smithi203.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:33:00.682 INFO:teuthology.orchestra.run.smithi145.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 37 kB/s | 22 kB 00:00 2024-03-29T11:33:00.891 INFO:teuthology.orchestra.run.smithi145.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 726 kB/s | 151 kB 00:00 2024-03-29T11:33:00.938 INFO:teuthology.orchestra.run.smithi203.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-29T11:33:00.949 INFO:teuthology.orchestra.run.smithi145.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 744 kB/s | 43 kB 00:00 2024-03-29T11:33:00.980 INFO:teuthology.orchestra.run.smithi203.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-29T11:33:01.058 INFO:teuthology.orchestra.run.smithi145.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-03-29T11:33:01.089 INFO:teuthology.orchestra.run.smithi203.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-29T11:33:01.160 INFO:teuthology.orchestra.run.smithi145.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.4 MB/s | 358 kB 00:00 2024-03-29T11:33:01.219 INFO:teuthology.orchestra.run.smithi145.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-03-29T11:33:01.278 INFO:teuthology.orchestra.run.smithi145.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-29T11:33:01.336 INFO:teuthology.orchestra.run.smithi145.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-03-29T11:33:01.395 INFO:teuthology.orchestra.run.smithi145.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 334 kB/s | 19 kB 00:00 2024-03-29T11:33:01.454 INFO:teuthology.orchestra.run.smithi145.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-03-29T11:33:01.504 INFO:teuthology.orchestra.run.smithi145.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 926 kB/s | 46 kB 00:00 2024-03-29T11:33:01.563 INFO:teuthology.orchestra.run.smithi145.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-29T11:33:01.564 INFO:teuthology.orchestra.run.smithi203.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 420 kB/s | 245 kB 00:00 2024-03-29T11:33:01.629 INFO:teuthology.orchestra.run.smithi145.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-29T11:33:01.688 INFO:teuthology.orchestra.run.smithi145.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-29T11:33:01.747 INFO:teuthology.orchestra.run.smithi145.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-29T11:33:01.805 INFO:teuthology.orchestra.run.smithi145.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2024-03-29T11:33:01.872 INFO:teuthology.orchestra.run.smithi145.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-29T11:33:01.948 INFO:teuthology.orchestra.run.smithi145.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-29T11:33:02.006 INFO:teuthology.orchestra.run.smithi145.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-03-29T11:33:05.023 INFO:teuthology.orchestra.run.smithi145.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 571 kB/s | 4.7 MB 00:08 2024-03-29T11:33:07.506 INFO:teuthology.orchestra.run.smithi203.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 1.0 MB/s | 7.4 MB 00:07 2024-03-29T11:33:07.508 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:07.508 INFO:teuthology.orchestra.run.smithi203.stdout:Total 4.8 MB/s | 37 MB 00:07 2024-03-29T11:33:07.552 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:07.577 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:07.577 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:07.944 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:07.945 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:08.643 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:08.720 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:08.752 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:08.844 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:08.880 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:09.058 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:09.095 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:09.185 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:09.347 INFO:teuthology.orchestra.run.smithi145.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 1.5 MB/s | 17 MB 00:11 2024-03-29T11:33:09.351 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:09.351 INFO:teuthology.orchestra.run.smithi145.stdout:Total 2.2 MB/s | 30 MB 00:13 2024-03-29T11:33:09.375 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:33:09.416 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:33:09.416 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:33:09.845 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:09.847 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:33:09.847 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:33:09.975 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:11.161 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:33:11.204 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:11.225 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:33:11.258 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:33:11.285 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:33:11.356 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:33:11.392 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:33:11.425 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:33:11.516 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:33:11.603 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:33:11.644 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:33:11.716 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:33:11.770 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:33:11.808 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:33:11.856 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:33:11.912 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:33:12.096 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:33:12.137 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:33:12.167 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:33:12.202 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:33:12.232 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:33:12.245 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:12.291 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:12.308 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:33:12.342 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:12.376 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:33:12.415 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:33:12.451 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:33:12.486 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:33:12.573 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:33:12.616 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:33:12.687 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:33:12.824 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:33:12.941 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:13.099 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:33:13.122 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:13.147 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:33:13.147 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:13.147 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:33:13.147 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:13.147 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:33:13.147 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:14.597 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:14.624 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:33:14.624 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:14.624 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:33:14.624 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:14.624 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:33:14.625 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:14.997 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:14.998 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:15.065 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:15.089 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:33:15.089 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:15.089 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:33:15.089 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:15.089 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:33:15.089 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:15.307 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:15.330 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:33:15.330 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:15.330 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:33:15.330 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:15.330 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:33:15.331 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:15.396 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:15.446 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:15.447 INFO:teuthology.orchestra.run.smithi203.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:15.448 INFO:teuthology.orchestra.run.smithi203.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:15.448 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:15.448 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:15.847 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:16.419 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:02 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:16.570 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:16.571 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-29T11:33:16.571 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:33:16.571 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-29T11:33:16.571 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:16.572 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:33:16.573 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-29T11:33:16.573 INFO:teuthology.orchestra.run.smithi203.stdout:Install 13 Packages 2024-03-29T11:33:16.573 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:16.573 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 2.0 M 2024-03-29T11:33:16.573 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 26 M 2024-03-29T11:33:16.573 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:33:16.845 INFO:teuthology.orchestra.run.smithi203.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 199 kB/s | 24 kB 00:00 2024-03-29T11:33:16.870 INFO:teuthology.orchestra.run.smithi203.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 122 kB/s | 18 kB 00:00 2024-03-29T11:33:16.920 INFO:teuthology.orchestra.run.smithi203.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:33:16.945 INFO:teuthology.orchestra.run.smithi203.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T11:33:16.971 INFO:teuthology.orchestra.run.smithi203.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:33:16.996 INFO:teuthology.orchestra.run.smithi203.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:33:17.021 INFO:teuthology.orchestra.run.smithi203.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 174 kB/s | 52 kB 00:00 2024-03-29T11:33:17.030 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:33:17.031 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:33:17.032 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:33:17.033 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:33:17.033 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:33:17.033 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:33:17.033 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:33:17.033 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:33:17.034 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:33:17.034 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:33:17.034 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:33:17.113 INFO:teuthology.orchestra.run.smithi203.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-29T11:33:17.139 INFO:teuthology.orchestra.run.smithi203.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-29T11:33:17.256 INFO:teuthology.orchestra.run.smithi203.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 866 kB/s | 203 kB 00:00 2024-03-29T11:33:17.281 INFO:teuthology.orchestra.run.smithi203.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 416 kB/s | 59 kB 00:00 2024-03-29T11:33:17.348 INFO:teuthology.orchestra.run.smithi203.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 973 kB/s | 90 kB 00:00 2024-03-29T11:33:17.398 INFO:teuthology.orchestra.run.smithi203.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.6 MB/s | 1.0 MB 00:00 2024-03-29T11:33:17.400 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:17.400 INFO:teuthology.orchestra.run.smithi203.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-29T11:33:17.401 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:17.415 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:17.415 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:17.512 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:17.512 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:17.936 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:33:17.936 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:17.936 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:33:17.936 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.936 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:33:17.937 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:33:17.938 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:33:17.939 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:33:17.939 INFO:teuthology.orchestra.run.smithi145.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:33:17.939 INFO:teuthology.orchestra.run.smithi145.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:33:17.939 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:33:17.939 INFO:teuthology.orchestra.run.smithi145.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:33:17.939 INFO:teuthology.orchestra.run.smithi145.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:33:17.939 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:17.939 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:33:18.035 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:18.203 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:18.266 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:18.316 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-base 2024-03-29T11:33:18.322 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:18.360 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:18.409 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:18.504 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:18.612 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:18.670 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:18.893 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:33:18.993 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:19.045 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:33:19.046 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-03-29T11:33:19.046 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:33:19.116 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install cephadm 2024-03-29T11:33:19.394 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:19.450 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:19.484 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:19.533 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:19.563 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:19.682 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:33:19.835 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:33:19.835 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:33:19.835 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 747 k 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 754 k 2024-03-29T11:33:19.836 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:33:20.139 INFO:teuthology.orchestra.run.smithi145.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.4 MB/s | 747 kB 00:00 2024-03-29T11:33:20.140 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:20.140 INFO:teuthology.orchestra.run.smithi145.stdout:Total 2.4 MB/s | 747 kB 00:00 2024-03-29T11:33:20.140 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:33:20.145 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:33:20.145 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:33:20.151 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:33:20.151 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:33:20.227 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:33:20.372 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:20.373 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:20.714 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:20.745 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:20.746 INFO:teuthology.orchestra.run.smithi203.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:20.747 INFO:teuthology.orchestra.run.smithi203.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:20.747 INFO:teuthology.orchestra.run.smithi203.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:20.747 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:20.747 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:20.819 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:21.010 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:21.202 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:21.490 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:33:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:21.935 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:33:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:21.935 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:21.935 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:33:21.937 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:07 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:21.937 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:21.938 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:21.939 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:21.939 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:21.939 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:33:21.939 INFO:teuthology.orchestra.run.smithi203.stdout:======================================================================================== 2024-03-29T11:33:21.939 INFO:teuthology.orchestra.run.smithi203.stdout:Install 6 Packages 2024-03-29T11:33:21.939 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:21.939 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 6.8 M 2024-03-29T11:33:21.939 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 33 M 2024-03-29T11:33:21.939 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:33:21.939 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:33:22.437 INFO:teuthology.orchestra.run.smithi203.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-29T11:33:22.519 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:33:22.537 INFO:teuthology.orchestra.run.smithi203.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T11:33:22.587 INFO:teuthology.orchestra.run.smithi203.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 432 kB/s | 138 kB 00:00 2024-03-29T11:33:22.679 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:22.680 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 145 k 2024-03-29T11:33:22.681 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 436 k 2024-03-29T11:33:22.681 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:33:22.863 INFO:teuthology.orchestra.run.smithi203.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 191 kB/s | 58 kB 00:00 2024-03-29T11:33:22.879 INFO:teuthology.orchestra.run.smithi203.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 175 kB/s | 51 kB 00:00 2024-03-29T11:33:22.980 INFO:teuthology.orchestra.run.smithi203.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-29T11:33:22.983 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:22.983 INFO:teuthology.orchestra.run.smithi203.stdout:Total 5.4 MB/s | 6.8 MB 00:01 2024-03-29T11:33:22.997 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:23.007 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:23.007 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:23.037 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 408 kB/s | 145 kB 00:00 2024-03-29T11:33:23.037 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:23.038 INFO:teuthology.orchestra.run.smithi145.stdout:Total 406 kB/s | 145 kB 00:00 2024-03-29T11:33:23.038 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:33:23.046 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:33:23.046 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:33:23.096 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:23.097 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:23.100 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:33:23.101 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:33:23.252 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:33:23.313 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:23.338 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:23.338 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:23.339 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:33:23.339 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:23.398 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:23.570 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:23.623 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:24.141 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:24.235 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:24.305 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:24.336 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:24.345 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:33:24.345 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:24.345 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:33:24.346 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:24.346 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:24.346 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:33:24.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr 2024-03-29T11:33:25.185 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:33:25.227 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:25.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:25.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:25.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:25.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:25.227 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:25.284 INFO:teuthology.orchestra.run.smithi145.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:25.337 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:33:25.338 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-03-29T11:33:25.338 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:33:25.413 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:33:25.543 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:25.543 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:25.543 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:25.543 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:25.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:25.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:25.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:25.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:25.543 INFO:teuthology.orchestra.run.smithi203.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:25.544 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:25.544 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:25.800 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-fuse 2024-03-29T11:33:25.978 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:01:52 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:33:26.130 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:33:26.131 INFO:teuthology.orchestra.run.smithi145.stdout:====================================================================================== 2024-03-29T11:33:26.131 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout:====================================================================================== 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:33:26.132 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:33:26.133 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:33:26.133 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:26.133 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:33:26.133 INFO:teuthology.orchestra.run.smithi145.stdout:====================================================================================== 2024-03-29T11:33:26.133 INFO:teuthology.orchestra.run.smithi145.stdout:Install 12 Packages 2024-03-29T11:33:26.133 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:26.134 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 6.9 M 2024-03-29T11:33:26.134 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 26 M 2024-03-29T11:33:26.134 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:33:26.365 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:26.516 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:26.516 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:26.516 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:33:26.516 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:26.516 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:33:26.516 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:33:26.516 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:33:26.516 INFO:teuthology.orchestra.run.smithi203.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:33:26.516 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:26.517 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:33:26.517 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:26.517 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-03-29T11:33:26.517 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:26.517 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 964 k 2024-03-29T11:33:26.517 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 2.8 M 2024-03-29T11:33:26.517 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:33:26.658 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 555 kB/s | 79 kB 00:00 2024-03-29T11:33:26.816 INFO:teuthology.orchestra.run.smithi145.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 162 kB/s | 31 kB 00:00 2024-03-29T11:33:26.841 INFO:teuthology.orchestra.run.smithi145.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 70 kB/s | 15 kB 00:00 2024-03-29T11:33:26.999 INFO:teuthology.orchestra.run.smithi145.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-29T11:33:27.033 INFO:teuthology.orchestra.run.smithi145.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-29T11:33:27.158 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 885 kB 00:00 2024-03-29T11:33:27.160 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:27.160 INFO:teuthology.orchestra.run.smithi203.stdout:Total 1.5 MB/s | 964 kB 00:00 2024-03-29T11:33:27.160 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:27.180 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:27.180 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:27.234 INFO:teuthology.orchestra.run.smithi145.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.9 MB/s | 1.8 MB 00:00 2024-03-29T11:33:27.254 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:27.254 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:27.400 INFO:teuthology.orchestra.run.smithi145.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-29T11:33:27.418 INFO:teuthology.orchestra.run.smithi145.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 106 kB/s | 19 kB 00:00 2024-03-29T11:33:27.436 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:27.443 INFO:teuthology.orchestra.run.smithi145.stdout:(8/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.9 MB/s | 863 kB 00:00 2024-03-29T11:33:27.577 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:33:27.601 INFO:teuthology.orchestra.run.smithi145.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 788 kB/s | 144 kB 00:00 2024-03-29T11:33:27.650 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:27.760 INFO:teuthology.orchestra.run.smithi145.stdout:(10/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.7 MB/s | 2.0 MB 00:00 2024-03-29T11:33:27.785 INFO:teuthology.orchestra.run.smithi145.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.0 MB/s | 188 kB 00:00 2024-03-29T11:33:28.329 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:33:28.329 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:33:28.644 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:33:28.644 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:28.644 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:28.644 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:33:28.644 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:28.644 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:28.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install ceph-volume 2024-03-29T11:33:29.505 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:29.660 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:29.660 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:29.660 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repository Size 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout:Installing dependencies: 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout:Install 2 Packages 2024-03-29T11:33:29.661 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:29.662 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 332 k 2024-03-29T11:33:29.662 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 1.5 M 2024-03-29T11:33:29.662 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:33:29.849 INFO:teuthology.orchestra.run.smithi203.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 370 kB/s | 69 kB 00:00 2024-03-29T11:33:30.115 INFO:teuthology.orchestra.run.smithi203.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 583 kB/s | 264 kB 00:00 2024-03-29T11:33:30.116 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:30.117 INFO:teuthology.orchestra.run.smithi203.stdout:Total 730 kB/s | 332 kB 00:00 2024-03-29T11:33:30.117 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:30.125 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:30.125 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:30.163 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:30.164 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:30.312 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:30.509 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:33:30.551 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:30.567 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:33:30.567 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:30.567 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:33:30.568 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:31.314 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:33:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:33:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:31.632 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:33:31.632 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:31.632 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:31.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados-devel 2024-03-29T11:33:32.564 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:32.714 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:32.715 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:32.715 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-29T11:33:32.715 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 127 k 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 456 k 2024-03-29T11:33:32.716 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:33:32.931 INFO:teuthology.orchestra.run.smithi203.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 590 kB/s | 127 kB 00:00 2024-03-29T11:33:32.931 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:32.931 INFO:teuthology.orchestra.run.smithi203.stdout:Total 587 kB/s | 127 kB 00:00 2024-03-29T11:33:32.932 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:32.937 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:32.937 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:32.993 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:32.994 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:33.142 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:33.256 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:33.702 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:34.005 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:34.005 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:34.005 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:34.005 INFO:teuthology.orchestra.run.smithi203.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:34.005 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:34.005 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:34.253 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs2 2024-03-29T11:33:34.643 INFO:teuthology.orchestra.run.smithi145.stdout:(12/12): python3-repoze-lru-0.7-16.el9.noarch.r 4.3 kB/s | 31 kB 00:07 2024-03-29T11:33:34.646 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:34.650 INFO:teuthology.orchestra.run.smithi145.stdout:Total 825 kB/s | 6.9 MB 00:08 2024-03-29T11:33:34.666 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:33:34.685 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:33:34.685 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:33:34.836 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:34.857 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:33:34.857 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:33:34.938 INFO:teuthology.orchestra.run.smithi203.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:34.992 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:34.993 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-29T11:33:34.993 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:35.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install libcephfs-devel 2024-03-29T11:33:35.371 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:33:35.459 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:33:35.495 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:33:35.540 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:33:35.580 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:33:35.647 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:35.686 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:33:35.802 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:35.803 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:35.803 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-29T11:33:35.803 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:35.803 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:33:35.803 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:33:35.804 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:35.804 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:33:35.804 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:35.804 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-29T11:33:35.804 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:35.804 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 31 k 2024-03-29T11:33:35.804 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 140 k 2024-03-29T11:33:35.805 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:33:35.805 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:33:35.890 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:33:35.942 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:33:35.971 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:35.986 INFO:teuthology.orchestra.run.smithi203.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 170 kB/s | 31 kB 00:00 2024-03-29T11:33:35.987 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:35.987 INFO:teuthology.orchestra.run.smithi203.stdout:Total 169 kB/s | 31 kB 00:00 2024-03-29T11:33:35.987 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:35.990 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:35.990 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:36.014 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:36.014 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:36.019 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:33:36.142 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:36.251 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:36.259 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:33:36.287 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:36.677 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:36.946 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:36.946 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:36.946 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:36.947 INFO:teuthology.orchestra.run.smithi203.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:36.947 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:36.947 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:37.127 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librados2 2024-03-29T11:33:37.192 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:33:37.193 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:33:37.541 INFO:teuthology.orchestra.run.smithi145.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:33:37.542 INFO:teuthology.orchestra.run.smithi145.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:33:37.542 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:33:37.542 INFO:teuthology.orchestra.run.smithi145.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:33:37.542 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:37.542 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:33:37.688 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:37.785 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:33:37.786 INFO:teuthology.orchestra.run.smithi203.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:37.836 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:37.837 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-29T11:33:37.837 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:37.882 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install librbd1 2024-03-29T11:33:38.349 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:05 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:33:38.448 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:38.503 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:33:38.504 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================ 2024-03-29T11:33:38.504 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:33:38.504 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================ 2024-03-29T11:33:38.504 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:33:38.504 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:33:38.504 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:33:38.504 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:33:38.504 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================================ 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout:Install 12 Packages 2024-03-29T11:33:38.505 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:38.506 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 37 M 2024-03-29T11:33:38.506 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 209 M 2024-03-29T11:33:38.506 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:33:38.550 INFO:teuthology.orchestra.run.smithi203.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:38.602 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:38.603 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-29T11:33:38.603 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:38.657 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rados 2024-03-29T11:33:38.924 INFO:teuthology.orchestra.run.smithi145.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 254 kB/s | 32 kB 00:00 2024-03-29T11:33:38.957 INFO:teuthology.orchestra.run.smithi145.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 517 kB/s | 17 kB 00:00 2024-03-29T11:33:38.999 INFO:teuthology.orchestra.run.smithi145.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 34 kB 00:00 2024-03-29T11:33:39.066 INFO:teuthology.orchestra.run.smithi145.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:33:39.223 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:39.321 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:39.334 INFO:teuthology.orchestra.run.smithi145.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-03-29T11:33:39.373 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-29T11:33:39.374 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:39.400 INFO:teuthology.orchestra.run.smithi145.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 4.8 MB 00:00 2024-03-29T11:33:39.424 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rgw 2024-03-29T11:33:39.426 INFO:teuthology.orchestra.run.smithi145.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:33:39.826 INFO:teuthology.orchestra.run.smithi145.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-29T11:33:39.991 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:40.089 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:40.141 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:40.142 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-29T11:33:40.142 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:40.196 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-cephfs 2024-03-29T11:33:40.759 INFO:teuthology.orchestra.run.smithi145.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 215 kB/s | 195 kB 00:00 2024-03-29T11:33:40.765 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:26 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:40.864 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:40.916 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:40.917 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-29T11:33:40.917 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:40.969 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install python3-rbd 2024-03-29T11:33:41.002 INFO:teuthology.orchestra.run.smithi145.stdout:(10/12): ceph-mgr-diskprediction-local-19.0.0-1 3.4 MB/s | 7.4 MB 00:02 2024-03-29T11:33:41.277 INFO:teuthology.orchestra.run.smithi145.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 434 kB/s | 803 kB 00:01 2024-03-29T11:33:41.536 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:41.635 INFO:teuthology.orchestra.run.smithi203.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:33:41.687 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:41.688 INFO:teuthology.orchestra.run.smithi203.stdout:Nothing to do. 2024-03-29T11:33:41.688 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:41.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-fuse 2024-03-29T11:33:41.751 INFO:teuthology.orchestra.run.smithi145.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 247 kB/s | 245 kB 00:00 2024-03-29T11:33:41.755 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:41.755 INFO:teuthology.orchestra.run.smithi145.stdout:Total 11 MB/s | 37 MB 00:03 2024-03-29T11:33:41.798 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:33:41.824 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:33:41.824 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:33:42.180 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:33:42.180 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:33:42.311 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:28 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:42.465 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:42.465 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:42.465 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:42.466 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:42.466 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:33:42.466 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:33:42.466 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:42.466 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:33:42.466 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:42.466 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-29T11:33:42.466 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:42.467 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 85 k 2024-03-29T11:33:42.467 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 231 k 2024-03-29T11:33:42.467 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:33:42.877 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:33:42.958 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:33:42.997 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:33:43.090 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:43.142 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:33:43.320 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:43.350 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:43.434 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:33:44.087 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:33:44.278 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:33:44.949 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 34 kB/s | 85 kB 00:02 2024-03-29T11:33:44.950 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:44.950 INFO:teuthology.orchestra.run.smithi203.stdout:Total 34 kB/s | 85 kB 00:02 2024-03-29T11:33:44.951 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:44.959 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:44.959 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:45.018 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:45.018 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:45.156 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:45.263 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:45.544 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:33:45.721 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:46.024 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:46.024 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:46.024 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:46.024 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:46.024 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:46.024 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:46.268 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-mirror 2024-03-29T11:33:46.578 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:33:46.615 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:46.837 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:32 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:46.991 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout: Package Arch Version Repo Size 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-29T11:33:46.992 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:46.993 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 3.1 M 2024-03-29T11:33:46.993 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 13 M 2024-03-29T11:33:46.993 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:33:47.570 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.4 MB/s | 3.1 MB 00:00 2024-03-29T11:33:47.571 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:47.571 INFO:teuthology.orchestra.run.smithi203.stdout:Total 5.3 MB/s | 3.1 MB 00:00 2024-03-29T11:33:47.571 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:47.580 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:47.580 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:47.654 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:47.654 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:48.079 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:48.146 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:48.171 INFO:teuthology.orchestra.run.smithi203.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:33:48.171 INFO:teuthology.orchestra.run.smithi203.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:33:48.171 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:48.171 INFO:teuthology.orchestra.run.smithi203.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:33:48.171 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:49.173 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:33:49.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:33:49.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:33:49.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:33:49.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:33:49.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:33:49.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:33:49.175 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:33:49.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:33:49.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:33:49.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:33:49.176 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:33:49.355 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:49.355 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:49.355 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:49.355 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:49.356 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:49.356 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:49.583 DEBUG:teuthology.orchestra.run.smithi203:> sudo yum -y install rbd-nbd 2024-03-29T11:33:49.601 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:49.602 INFO:teuthology.orchestra.run.smithi145.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:33:49.603 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:33:49.603 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:49.603 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:33:49.973 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:33:50.161 INFO:teuthology.orchestra.run.smithi203.stdout:Last metadata expiration check: 0:02:36 ago on Fri 29 Mar 2024 11:31:14 AM UTC. 2024-03-29T11:33:50.309 INFO:teuthology.orchestra.run.smithi203.stdout:Dependencies resolved. 2024-03-29T11:33:50.310 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:50.310 INFO:teuthology.orchestra.run.smithi203.stdout: Package Architecture Version Repository Size 2024-03-29T11:33:50.310 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:50.310 INFO:teuthology.orchestra.run.smithi203.stdout:Installing: 2024-03-29T11:33:50.310 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:33:50.311 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:50.311 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction Summary 2024-03-29T11:33:50.311 INFO:teuthology.orchestra.run.smithi203.stdout:================================================================================ 2024-03-29T11:33:50.311 INFO:teuthology.orchestra.run.smithi203.stdout:Install 1 Package 2024-03-29T11:33:50.311 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:50.311 INFO:teuthology.orchestra.run.smithi203.stdout:Total download size: 171 k 2024-03-29T11:33:50.311 INFO:teuthology.orchestra.run.smithi203.stdout:Installed size: 494 k 2024-03-29T11:33:50.311 INFO:teuthology.orchestra.run.smithi203.stdout:Downloading Packages: 2024-03-29T11:33:50.546 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:33:50.667 INFO:teuthology.orchestra.run.smithi203.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 482 kB/s | 171 kB 00:00 2024-03-29T11:33:50.668 INFO:teuthology.orchestra.run.smithi203.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:50.668 INFO:teuthology.orchestra.run.smithi203.stdout:Total 480 kB/s | 171 kB 00:00 2024-03-29T11:33:50.668 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction check 2024-03-29T11:33:50.676 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction check succeeded. 2024-03-29T11:33:50.676 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction test 2024-03-29T11:33:50.697 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================== 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================== 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:33:50.699 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================== 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout:Install 13 Packages 2024-03-29T11:33:50.700 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:50.701 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 2.0 M 2024-03-29T11:33:50.701 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 26 M 2024-03-29T11:33:50.701 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:33:50.733 INFO:teuthology.orchestra.run.smithi203.stdout:Transaction test succeeded. 2024-03-29T11:33:50.733 INFO:teuthology.orchestra.run.smithi203.stdout:Running transaction 2024-03-29T11:33:50.878 INFO:teuthology.orchestra.run.smithi203.stdout: Preparing : 1/1 2024-03-29T11:33:50.932 INFO:teuthology.orchestra.run.smithi145.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 200 kB/s | 24 kB 00:00 2024-03-29T11:33:50.957 INFO:teuthology.orchestra.run.smithi145.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 123 kB/s | 18 kB 00:00 2024-03-29T11:33:50.982 INFO:teuthology.orchestra.run.smithi203.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:51.024 INFO:teuthology.orchestra.run.smithi145.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:33:51.050 INFO:teuthology.orchestra.run.smithi145.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-29T11:33:51.075 INFO:teuthology.orchestra.run.smithi145.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:33:51.109 INFO:teuthology.orchestra.run.smithi145.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 735 kB/s | 43 kB 00:00 2024-03-29T11:33:51.151 INFO:teuthology.orchestra.run.smithi145.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 153 kB/s | 52 kB 00:00 2024-03-29T11:33:51.343 INFO:teuthology.orchestra.run.smithi145.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-29T11:33:51.376 INFO:teuthology.orchestra.run.smithi145.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-29T11:33:51.417 INFO:teuthology.orchestra.run.smithi203.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:51.585 INFO:teuthology.orchestra.run.smithi145.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 467 kB/s | 203 kB 00:00 2024-03-29T11:33:51.695 INFO:teuthology.orchestra.run.smithi203.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:33:51.696 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:51.696 INFO:teuthology.orchestra.run.smithi203.stdout:Installed: 2024-03-29T11:33:51.696 INFO:teuthology.orchestra.run.smithi203.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:33:51.697 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:33:51.697 INFO:teuthology.orchestra.run.smithi203.stdout:Complete! 2024-03-29T11:33:51.735 INFO:teuthology.orchestra.run.smithi145.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 165 kB/s | 59 kB 00:00 2024-03-29T11:33:51.921 DEBUG:teuthology.parallel:result is None 2024-03-29T11:33:51.927 INFO:teuthology.orchestra.run.smithi145.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 262 kB/s | 90 kB 00:00 2024-03-29T11:33:52.094 INFO:teuthology.orchestra.run.smithi145.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2024-03-29T11:33:52.096 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:52.097 INFO:teuthology.orchestra.run.smithi145.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-29T11:33:52.097 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:33:52.112 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:33:52.112 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:33:52.209 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:33:52.209 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:33:52.726 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:33:52.887 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:33:52.944 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:33:52.999 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:33:53.037 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:33:53.078 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:33:53.170 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:33:53.279 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:33:53.329 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:33:54.042 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:33:54.110 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:33:54.144 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:33:54.192 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:33:54.214 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:33:55.060 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:33:55.061 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:33:55.061 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:33:55.061 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:33:55.418 INFO:teuthology.orchestra.run.smithi145.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:33:55.419 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:33:55.419 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:33:55.419 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:33:55.419 INFO:teuthology.orchestra.run.smithi145.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:33:55.419 INFO:teuthology.orchestra.run.smithi145.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:33:55.419 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:55.419 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:33:55.668 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:33:56.252 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:33:56.405 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:33:56.406 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================== 2024-03-29T11:33:56.406 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:33:56.406 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================== 2024-03-29T11:33:56.407 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:33:56.407 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:33:56.407 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:33:56.407 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:33:56.407 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:33:56.408 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:33:56.408 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:33:56.408 INFO:teuthology.orchestra.run.smithi145.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:33:56.408 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:56.409 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:33:56.409 INFO:teuthology.orchestra.run.smithi145.stdout:======================================================================================== 2024-03-29T11:33:56.409 INFO:teuthology.orchestra.run.smithi145.stdout:Install 6 Packages 2024-03-29T11:33:56.409 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:56.410 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 6.8 M 2024-03-29T11:33:56.411 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 33 M 2024-03-29T11:33:56.411 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:33:56.762 INFO:teuthology.orchestra.run.smithi145.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T11:33:57.012 INFO:teuthology.orchestra.run.smithi145.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-03-29T11:33:57.038 INFO:teuthology.orchestra.run.smithi145.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 313 kB/s | 138 kB 00:00 2024-03-29T11:33:57.346 INFO:teuthology.orchestra.run.smithi145.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 173 kB/s | 58 kB 00:00 2024-03-29T11:33:57.372 INFO:teuthology.orchestra.run.smithi145.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 901 kB/s | 548 kB 00:00 2024-03-29T11:33:57.388 INFO:teuthology.orchestra.run.smithi145.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 145 kB/s | 51 kB 00:00 2024-03-29T11:33:57.390 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:33:57.390 INFO:teuthology.orchestra.run.smithi145.stdout:Total 6.9 MB/s | 6.8 MB 00:00 2024-03-29T11:33:57.404 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:33:57.414 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:33:57.414 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:33:57.503 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:33:57.504 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:33:57.820 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:33:57.998 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:33:58.065 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:33:58.580 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:33:58.659 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:33:58.728 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:33:58.752 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:59.621 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:33:59.621 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:33:59.621 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:33:59.621 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:33:59.621 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:33:59.621 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:33:59.922 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:33:59.922 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:59.922 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:33:59.923 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:33:59.923 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:33:59.923 INFO:teuthology.orchestra.run.smithi145.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:33:59.923 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:33:59.923 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:33:59.923 INFO:teuthology.orchestra.run.smithi145.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:33:59.923 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:33:59.923 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:00.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-fuse 2024-03-29T11:34:00.706 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:00.858 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:00.858 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:00.858 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout:Install 2 Packages 2024-03-29T11:34:00.859 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:00.860 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 964 k 2024-03-29T11:34:00.860 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 2.8 M 2024-03-29T11:34:00.860 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:34:00.981 INFO:teuthology.orchestra.run.smithi145.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 648 kB/s | 79 kB 00:00 2024-03-29T11:34:01.298 INFO:teuthology.orchestra.run.smithi145.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.0 MB/s | 885 kB 00:00 2024-03-29T11:34:01.299 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:01.300 INFO:teuthology.orchestra.run.smithi145.stdout:Total 2.1 MB/s | 964 kB 00:00 2024-03-29T11:34:01.300 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:34:01.321 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:34:01.321 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:34:01.394 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:34:01.395 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:34:01.577 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:34:01.732 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:34:01.815 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:34:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:34:02.529 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:34:02.839 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:34:02.839 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:02.839 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:34:02.839 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:34:02.839 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:02.839 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:03.128 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install ceph-volume 2024-03-29T11:34:03.710 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:30 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:03.863 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:03.863 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:03.863 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repository Size 2024-03-29T11:34:03.863 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:03.863 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:34:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:34:03.864 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-03-29T11:34:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:34:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:03.864 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:34:03.864 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:03.864 INFO:teuthology.orchestra.run.smithi145.stdout:Install 2 Packages 2024-03-29T11:34:03.864 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:03.865 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 332 k 2024-03-29T11:34:03.865 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 1.5 M 2024-03-29T11:34:03.865 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:34:03.986 INFO:teuthology.orchestra.run.smithi145.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 568 kB/s | 69 kB 00:00 2024-03-29T11:34:04.136 INFO:teuthology.orchestra.run.smithi145.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 972 kB/s | 264 kB 00:00 2024-03-29T11:34:04.137 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:04.138 INFO:teuthology.orchestra.run.smithi145.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-29T11:34:04.138 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:34:04.147 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:34:04.147 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:34:04.182 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:34:04.183 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:34:04.329 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:34:04.494 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:34:04.539 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:34:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:34:04.555 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:04.555 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:34:04.555 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:05.265 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:34:05.613 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:34:05.613 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:05.613 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:34:05.613 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:34:05.613 INFO:teuthology.orchestra.run.smithi145.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:34:05.613 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:05.613 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:05.940 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librados-devel 2024-03-29T11:34:06.537 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:33 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:06.691 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:06.692 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:06.692 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2024-03-29T11:34:06.692 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:06.692 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:34:06.692 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:34:06.693 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:06.693 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:34:06.693 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:06.693 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-03-29T11:34:06.693 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:06.693 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 127 k 2024-03-29T11:34:06.693 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 456 k 2024-03-29T11:34:06.693 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:34:06.927 INFO:teuthology.orchestra.run.smithi145.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 541 kB/s | 127 kB 00:00 2024-03-29T11:34:06.927 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:06.928 INFO:teuthology.orchestra.run.smithi145.stdout:Total 538 kB/s | 127 kB 00:00 2024-03-29T11:34:06.928 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:34:06.933 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:34:06.933 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:34:06.989 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:34:06.989 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:34:07.132 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:34:07.235 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:07.718 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:08.011 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:34:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:08.011 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:08.011 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:08.255 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs2 2024-03-29T11:34:08.835 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:35 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:08.938 INFO:teuthology.orchestra.run.smithi145.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:08.993 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:08.994 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-03-29T11:34:08.994 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:09.069 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install libcephfs-devel 2024-03-29T11:34:09.639 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:36 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:09.791 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:09.792 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:09.792 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2024-03-29T11:34:09.792 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:09.792 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:34:09.792 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:34:09.792 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:09.792 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:34:09.793 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:09.793 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-03-29T11:34:09.793 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:09.793 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 31 k 2024-03-29T11:34:09.793 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 140 k 2024-03-29T11:34:09.793 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:34:09.955 INFO:teuthology.orchestra.run.smithi145.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 192 kB/s | 31 kB 00:00 2024-03-29T11:34:09.956 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:09.956 INFO:teuthology.orchestra.run.smithi145.stdout:Total 191 kB/s | 31 kB 00:00 2024-03-29T11:34:09.956 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:34:09.959 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:34:09.959 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:34:09.983 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:34:09.984 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:34:10.102 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:34:10.228 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:10.601 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:10.861 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:10.861 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:10.861 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:34:10.861 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:10.861 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:10.861 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:11.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librados2 2024-03-29T11:34:11.572 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:38 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:11.678 INFO:teuthology.orchestra.run.smithi145.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:11.721 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:11.733 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-03-29T11:34:11.733 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:11.769 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install librbd1 2024-03-29T11:34:12.332 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:39 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:12.429 INFO:teuthology.orchestra.run.smithi145.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:12.482 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:12.483 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-03-29T11:34:12.483 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:12.530 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rados 2024-03-29T11:34:13.100 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:13.199 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:13.252 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:13.252 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-03-29T11:34:13.252 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:13.295 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rgw 2024-03-29T11:34:13.858 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:13.958 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:14.009 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:14.010 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-03-29T11:34:14.010 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:14.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-cephfs 2024-03-29T11:34:14.625 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:41 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:14.723 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:14.775 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:14.776 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-03-29T11:34:14.776 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:14.819 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install python3-rbd 2024-03-29T11:34:15.385 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:42 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:15.484 INFO:teuthology.orchestra.run.smithi145.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:34:15.536 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:15.537 INFO:teuthology.orchestra.run.smithi145.stdout:Nothing to do. 2024-03-29T11:34:15.537 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:15.582 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-fuse 2024-03-29T11:34:16.142 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:43 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:16.290 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:16.291 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 85 k 2024-03-29T11:34:16.292 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 231 k 2024-03-29T11:34:16.292 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:34:16.650 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 238 kB/s | 85 kB 00:00 2024-03-29T11:34:16.651 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:16.651 INFO:teuthology.orchestra.run.smithi145.stdout:Total 237 kB/s | 85 kB 00:00 2024-03-29T11:34:16.651 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:34:16.658 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:34:16.658 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:34:16.713 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:34:16.713 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:34:16.850 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:34:16.965 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:17.381 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:17.677 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:34:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:17.677 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:17.677 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:17.925 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-mirror 2024-03-29T11:34:18.502 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:45 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:18.657 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:18.658 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:18.658 INFO:teuthology.orchestra.run.smithi145.stdout: Package Arch Version Repo Size 2024-03-29T11:34:18.658 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:18.658 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:34:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:34:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:18.659 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:34:18.659 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:18.659 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-03-29T11:34:18.659 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:18.659 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 3.1 M 2024-03-29T11:34:18.659 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 13 M 2024-03-29T11:34:18.659 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:34:19.136 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 6.5 MB/s | 3.1 MB 00:00 2024-03-29T11:34:19.136 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:19.137 INFO:teuthology.orchestra.run.smithi145.stdout:Total 6.5 MB/s | 3.1 MB 00:00 2024-03-29T11:34:19.137 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:34:19.146 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:34:19.146 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:34:19.219 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:34:19.220 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:34:19.649 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:34:19.720 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:19.744 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:19.744 INFO:teuthology.orchestra.run.smithi145.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:34:19.744 INFO:teuthology.orchestra.run.smithi145.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:34:19.744 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:19.744 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:34:19.744 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:20.877 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:20.878 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:34:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:20.878 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:20.878 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:21.098 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum -y install rbd-nbd 2024-03-29T11:34:21.668 INFO:teuthology.orchestra.run.smithi145.stdout:Last metadata expiration check: 0:02:48 ago on Fri 29 Mar 2024 11:31:33 AM UTC. 2024-03-29T11:34:21.819 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-03-29T11:34:21.820 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:21.820 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-03-29T11:34:21.820 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:21.820 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-03-29T11:34:21.820 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:34:21.820 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:21.820 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-03-29T11:34:21.821 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-03-29T11:34:21.821 INFO:teuthology.orchestra.run.smithi145.stdout:Install 1 Package 2024-03-29T11:34:21.821 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:21.821 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 171 k 2024-03-29T11:34:21.821 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 494 k 2024-03-29T11:34:21.821 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-03-29T11:34:22.239 INFO:teuthology.orchestra.run.smithi145.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 410 kB/s | 171 kB 00:00 2024-03-29T11:34:22.239 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:34:22.240 INFO:teuthology.orchestra.run.smithi145.stdout:Total 409 kB/s | 171 kB 00:00 2024-03-29T11:34:22.240 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-03-29T11:34:22.247 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-03-29T11:34:22.248 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-03-29T11:34:22.302 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-03-29T11:34:22.303 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-03-29T11:34:22.445 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-03-29T11:34:22.562 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:22.994 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:23.310 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:34:23.310 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:23.310 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-03-29T11:34:23.311 INFO:teuthology.orchestra.run.smithi145.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:34:23.311 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:23.311 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-03-29T11:34:23.538 DEBUG:teuthology.parallel:result is None 2024-03-29T11:34:23.539 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:34:23.676 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:34:23.697 INFO:teuthology.orchestra.run.smithi145.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:23.698 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:23.698 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:34:23.700 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:34:23.937 DEBUG:teuthology.orchestra.run.smithi203:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:34:23.962 INFO:teuthology.orchestra.run.smithi203.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:23.962 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:34:23.962 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:34:23.964 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:34:23.964 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:23.964 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:34:23.992 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:23.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:34:24.034 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:34:24.034 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:24.034 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:34:24.061 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:34:24.128 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:24.128 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:34:24.158 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:34:24.227 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:34:24.227 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:24.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:34:24.254 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:34:24.322 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:24.322 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:34:24.351 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:34:24.418 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:34:24.419 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:24.419 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:34:24.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:34:24.513 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:24.513 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:34:24.540 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:34:24.607 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:34:24.696 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:34:24.696 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 777 /var/log/ceph 2024-03-29T11:34:24.699 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 777 /var/log/ceph 2024-03-29T11:34:24.726 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:34:24.726 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:34:24.768 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:34:24.795 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:34:24.795 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:34:24.834 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:34:24.873 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:34:24.874 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:34:24.874 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622713', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi145.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPiPMOp+vgiEZZNa8LPYohHLQRxJY9R6LGA1gZHjclHL8fx7cA2Mxw4cYFT8DwlUftEIvyJowLK2lvMRhM8DeK8=', 'smithi203.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZDVBhzAjM1D8MQiaC92e7nFlm/phBSL2LVW0XsOysQ2+y1EDEUrkFPDQ5iXfuu8mbjppQok81+OL7WRQ6WzLs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:34:24.874 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:34:24.903 DEBUG:teuthology.orchestra.run.smithi203:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:34:24.918 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:34:24.957 DEBUG:teuthology.orchestra.run.smithi203:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:34:24.986 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:24.987 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:34:25.037 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:34:25.038 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 2024-03-29T11:34:25.092 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:34:25.092 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:25.092 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-29T11:34:25.092 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:25.092 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:25.092 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-29 11:34:19.984288831 +0000 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-29 11:30:30.687451185 +0000 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-29 11:30:30.687451185 +0000 2024-03-29T11:34:25.093 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-29 11:30:30.687451185 +0000 2024-03-29T11:34:25.093 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:34:25.158 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-29T11:34:25.158 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-29T11:34:25.158 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000237862 s, 2.2 MB/s 2024-03-29T11:34:25.159 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:34:25.215 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 2024-03-29T11:34:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:34:25.269 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:25.269 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:34:25.270 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:25.270 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:25.270 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-29 11:34:19.984288831 +0000 2024-03-29T11:34:25.270 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-29 11:30:31.046443382 +0000 2024-03-29T11:34:25.270 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-29 11:30:31.046443382 +0000 2024-03-29T11:34:25.270 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-29 11:30:31.046443382 +0000 2024-03-29T11:34:25.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:34:25.334 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-29T11:34:25.334 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-29T11:34:25.334 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000184856 s, 2.8 MB/s 2024-03-29T11:34:25.335 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:34:25.392 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 2024-03-29T11:34:25.447 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:34:25.447 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:25.447 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-29T11:34:25.447 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:25.447 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:25.447 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-29 11:34:19.985288810 +0000 2024-03-29T11:34:25.447 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-29 11:30:31.396435774 +0000 2024-03-29T11:34:25.447 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-29 11:30:31.396435774 +0000 2024-03-29T11:34:25.447 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-29 11:30:31.396435774 +0000 2024-03-29T11:34:25.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:34:25.513 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-29T11:34:25.513 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-29T11:34:25.513 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.00021932 s, 2.3 MB/s 2024-03-29T11:34:25.514 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:34:25.570 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 2024-03-29T11:34:25.625 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:34:25.625 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:25.625 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-29T11:34:25.625 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:25.625 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:25.626 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-03-29 11:34:19.985288810 +0000 2024-03-29T11:34:25.626 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-03-29 11:30:31.731428493 +0000 2024-03-29T11:34:25.626 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-03-29 11:30:31.731428493 +0000 2024-03-29T11:34:25.626 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-03-29 11:30:31.731428493 +0000 2024-03-29T11:34:25.626 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:34:25.691 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-03-29T11:34:25.691 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-03-29T11:34:25.691 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000208736 s, 2.5 MB/s 2024-03-29T11:34:25.692 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:34:25.748 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:25.748 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:25.748 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:34:25.763 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:34:25.764 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_1 2024-03-29T11:34:25.819 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:34:25.819 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:25.819 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:34:25.819 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:25.819 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:25.819 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-29 11:33:48.422114779 +0000 2024-03-29T11:34:25.819 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-29 11:30:22.680922012 +0000 2024-03-29T11:34:25.819 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-29 11:30:22.680922012 +0000 2024-03-29T11:34:25.819 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-29 11:30:22.680922012 +0000 2024-03-29T11:34:25.820 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:34:25.884 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-29T11:34:25.885 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-29T11:34:25.885 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000202387 s, 2.5 MB/s 2024-03-29T11:34:25.886 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:34:25.942 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_2 2024-03-29T11:34:25.997 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:34:25.998 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:25.998 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-29T11:34:25.998 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:25.998 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:25.998 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-29 11:33:48.422114779 +0000 2024-03-29T11:34:25.998 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-29 11:30:23.022913415 +0000 2024-03-29T11:34:25.998 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-29 11:30:23.022913415 +0000 2024-03-29T11:34:25.998 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-29 11:30:23.022913415 +0000 2024-03-29T11:34:25.998 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:34:26.064 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-29T11:34:26.064 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-29T11:34:26.064 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000198062 s, 2.6 MB/s 2024-03-29T11:34:26.065 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:34:26.120 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_3 2024-03-29T11:34:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:34:26.174 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:26.174 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:34:26.174 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:26.174 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:26.174 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-29 11:33:48.423114754 +0000 2024-03-29T11:34:26.174 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-29 11:30:23.538900444 +0000 2024-03-29T11:34:26.175 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-29 11:30:23.538900444 +0000 2024-03-29T11:34:26.175 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-29 11:30:23.538900444 +0000 2024-03-29T11:34:26.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:34:26.239 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-29T11:34:26.239 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-29T11:34:26.239 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.000223662 s, 2.3 MB/s 2024-03-29T11:34:26.241 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:34:26.296 DEBUG:teuthology.orchestra.run.smithi203:> stat /dev/vg_nvme/lv_4 2024-03-29T11:34:26.351 INFO:teuthology.orchestra.run.smithi203.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:34:26.351 INFO:teuthology.orchestra.run.smithi203.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:34:26.352 INFO:teuthology.orchestra.run.smithi203.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:34:26.352 INFO:teuthology.orchestra.run.smithi203.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:34:26.352 INFO:teuthology.orchestra.run.smithi203.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:34:26.352 INFO:teuthology.orchestra.run.smithi203.stdout:Access: 2024-03-29 11:33:48.423114754 +0000 2024-03-29T11:34:26.352 INFO:teuthology.orchestra.run.smithi203.stdout:Modify: 2024-03-29 11:30:23.881891822 +0000 2024-03-29T11:34:26.352 INFO:teuthology.orchestra.run.smithi203.stdout:Change: 2024-03-29 11:30:23.881891822 +0000 2024-03-29T11:34:26.352 INFO:teuthology.orchestra.run.smithi203.stdout: Birth: 2024-03-29 11:30:23.881891822 +0000 2024-03-29T11:34:26.352 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:34:26.417 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records in 2024-03-29T11:34:26.417 INFO:teuthology.orchestra.run.smithi203.stderr:1+0 records out 2024-03-29T11:34:26.418 INFO:teuthology.orchestra.run.smithi203.stderr:512 bytes copied, 0.00020795 s, 2.5 MB/s 2024-03-29T11:34:26.419 DEBUG:teuthology.orchestra.run.smithi203:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:34:26.474 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:26.474 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi145.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:34:26.474 INFO:tasks.ceph:Generating config... 2024-03-29T11:34:26.476 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:34:26.476 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:34:26.476 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:34:26.476 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:34:26.476 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:34:26.476 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:34:26.476 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:34:26.476 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:34:26.476 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:34:26.476 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:34:26.476 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:34:26.476 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:34:26.477 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:34:26.477 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:34:26.477 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:34:26.477 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:34:26.478 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:34:26.479 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:34:26.479 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:34:26.480 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:26.540 INFO:teuthology.orchestra.run.smithi145.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:34:26.542 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:34:26.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:26.614 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.145:6789]'), ('mon.c', '[v1:172.21.15.145:6790]'), ('mon.b', '[v1:172.21.15.203:6789]')] 2024-03-29T11:34:26.615 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.145:6789],[v1:172.21.15.145:6790],[v1:172.21.15.203:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:34:26.616 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:26.616 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:34:26.669 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.145:6789]' --addv c '[v1:172.21.15.145:6790]' --addv b '[v1:172.21.15.203:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:26.754 INFO:teuthology.orchestra.run.smithi145.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:26.754 INFO:teuthology.orchestra.run.smithi145.stdout:monmaptool: generated fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:setting min_mon_release = quincy 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:epoch 0 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:min_mon_release 17 (quincy) 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:election_strategy: 1 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:34:26.755 INFO:teuthology.orchestra.run.smithi145.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:34:26.756 DEBUG:teuthology.orchestra.run.smithi145:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:34:26.772 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 524911ae-c3df-44f6-b399-af269c927521... 2024-03-29T11:34:26.772 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:34:26.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:34:26.862 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: chdir = "" 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: auth supported = cephx 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: filestore xattr use omap = true 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: mon clock drift allowed = 1.000 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: osd crush chooseleaf type = 0 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: auth debug = true 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on old message = true 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on bug = true 2024-03-29T11:34:26.863 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon pg warn max object skew = 0 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default size = 2 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow primary affinity = true 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow pg remap = true 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on no sortbitwise = false 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:34:26.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on too few osds = false 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon_allow_pool_size_one = true 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: osd default data pool replay window = 5 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon allow pool delete = true 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon cluster log file level = debug 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: debug asserts on shutdown = true 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon health detail to clog = false 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon host = "[v1:172.21.15.145:6789],[v1:172.21.15.145:6790],[v1:172.21.15.203:6789]" 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: lockdep = True 2024-03-29T11:34:26.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon client directed command retry = 5 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: mon election default strategy = 1 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: ms bind msgr2 = False 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: ms inject socket failures = 5000 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: ms type = async 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout:[osd] 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: osd journal size = 100 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub load threshold = 5.0 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub max interval = 600 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.866 INFO:teuthology.orchestra.run.smithi145.stdout: osd recover clone overlap = true 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd recovery max chunk = 1048576 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug shutdown = true 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug op order = true 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify stray on activate = true 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug trim objects = true 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd open classes on start = true 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug pg log writeout = true 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd map max advance = 10 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.867 INFO:teuthology.orchestra.run.smithi145.stdout: journal zero on create = true 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: filestore apply finisher threads = 3 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: bdev debug aio = true 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug misdirected ops = true 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore block size = 96636764160 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore compression algorithm = snappy 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore compression mode = aggressive 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore fsck on mount = True 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore zero block detection = True 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluefs = 20 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluestore = 20 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: debug monc = 20 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-03-29T11:34:26.868 INFO:teuthology.orchestra.run.smithi145.stdout: debug osd = 20 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: debug rocksdb = 10 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd full ratio = 0.9 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd class default list = * 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd class load list = * 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd client watch timeout = 120 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify cached snaps = True 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify missing on start = True 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd mclock override recovery settings = True 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd objectstore = bluestore 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd op queue = debug_random 2024-03-29T11:34:26.869 INFO:teuthology.orchestra.run.smithi145.stdout: osd op queue cut off = debug_random 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: osd shutdown pgref assert = True 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr] 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: debug mgr = 20 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: mgr/telemetry/nag = false 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout:[mon] 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-03-29T11:34:26.870 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: debug paxos = 20 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: mon data avail warn = 5 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: mon mgr mkfs grace = 240 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd reporter subtree level = osd 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd prime pg temp = true 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:34:26.871 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: # 1m isn't quite enough 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: mon min osdmap epochs = 50 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: mon osdmap full prune interval = 2 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: mon osdmap full prune min = 15 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: mon scrub interval = 300 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on pool no app = False 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: paxos service trim min = 10 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout:[client] 2024-03-29T11:34:26.872 INFO:teuthology.orchestra.run.smithi145.stdout: rgw cache enabled = true 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable ops log = true 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable usage log = true 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout: debug monc = 20 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout: debug objecter = 20 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout: debug rados = 20 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.a] 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.c] 2024-03-29T11:34:26.873 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.b] 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout:[global] 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: chdir = "" 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: auth supported = cephx 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: filestore xattr use omap = true 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: mon clock drift allowed = 1.000 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: osd crush chooseleaf type = 0 2024-03-29T11:34:26.874 INFO:teuthology.orchestra.run.smithi203.stdout: auth debug = true 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on old message = true 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: ms die on bug = true 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: mon pg warn max object skew = 0 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default size = 2 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow primary affinity = true 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd allow pg remap = true 2024-03-29T11:34:26.875 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on no sortbitwise = false 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on too few osds = false 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: mon_allow_pool_size_one = true 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: osd default data pool replay window = 5 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: mon allow pool delete = true 2024-03-29T11:34:26.876 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: mon cluster log file level = debug 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: debug asserts on shutdown = true 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: mon health detail to clog = false 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: mon host = "[v1:172.21.15.145:6789],[v1:172.21.15.145:6790],[v1:172.21.15.203:6789]" 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: lockdep = True 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: mon client directed command retry = 5 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: mon election default strategy = 1 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: ms bind msgr2 = False 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: ms inject socket failures = 5000 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: ms type = async 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: fsid = 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout:[osd] 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: osd journal size = 100 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub load threshold = 5.0 2024-03-29T11:34:26.877 INFO:teuthology.orchestra.run.smithi203.stdout: osd scrub max interval = 600 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd recover clone overlap = true 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd recovery max chunk = 1048576 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug shutdown = true 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug op order = true 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify stray on activate = true 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug trim objects = true 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd open classes on start = true 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug pg log writeout = true 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:34:26.878 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: osd map max advance = 10 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: journal zero on create = true 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: filestore apply finisher threads = 3 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: bdev debug aio = true 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug misdirected ops = true 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore block size = 96636764160 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore compression algorithm = snappy 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore compression mode = aggressive 2024-03-29T11:34:26.879 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore fsck on mount = True 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: bluestore zero block detection = True 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluefs = 20 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: debug bluestore = 20 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: debug osd = 20 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: debug rocksdb = 10 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd full ratio = 0.9 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: osd class default list = * 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: osd class load list = * 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: osd client watch timeout = 120 2024-03-29T11:34:26.880 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify cached snaps = True 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd debug verify missing on start = True 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd mclock override recovery settings = True 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd objectstore = bluestore 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue = debug_random 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd op queue cut off = debug_random 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: osd shutdown pgref assert = True 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr] 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: debug mgr = 20 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:26.881 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: mgr/telemetry/nag = false 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout:[mon] 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: debug mon = 20 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: debug paxos = 20 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: debug auth = 20 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: mon data avail warn = 5 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: mon mgr mkfs grace = 240 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd reporter subtree level = osd 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: mon osd prime pg temp = true 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:34:26.882 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: # 1m isn't quite enough 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon min osdmap epochs = 50 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune interval = 2 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune min = 15 2024-03-29T11:34:26.883 INFO:teuthology.orchestra.run.smithi203.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: mon scrub interval = 300 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: mon warn on pool no app = False 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: paxos service trim min = 10 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout:[client] 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: rgw cache enabled = true 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable ops log = true 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: rgw enable usage log = true 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: debug monc = 20 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: debug ms = 1 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: debug objecter = 20 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout: debug rados = 20 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.a] 2024-03-29T11:34:26.884 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.c] 2024-03-29T11:34:26.885 INFO:teuthology.orchestra.run.smithi203.stdout:[mon.b] 2024-03-29T11:34:26.886 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:34:26.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:34:26.941 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:34:26.941 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:26.941 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:34:26.956 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:26.956 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:34:27.011 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:34:27.011 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:27.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:34:27.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:27.090 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:27.091 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:27.145 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:34:27.145 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:27.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:34:27.145 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:34:27.185 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:27.185 DEBUG:teuthology.orchestra.run.smithi203:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:27.241 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:34:27.241 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:34:27.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:34:27.296 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:34:27.298 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:34:27.365 INFO:teuthology.orchestra.run.smithi203.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:34:27.368 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:34:27.368 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:34:27.368 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:34:27.410 INFO:teuthology.orchestra.run.smithi145.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:34:27.425 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:34:27.467 INFO:teuthology.orchestra.run.smithi203.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:34:27.482 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:34:27.483 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi145.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi203.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:34:27.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:27.509 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:27.509 INFO:tasks.ceph:role: osd.0 2024-03-29T11:34:27.509 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:34:27.510 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:27.583 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:27.584 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:27.584 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:27.584 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:27.584 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:27.584 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:27.584 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:27.584 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:27.584 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:27.584 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:28.084 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-03-29T11:34:28.086 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-03-29T11:34:28.086 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:28.191 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:28.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:28.294 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:28.294 INFO:tasks.ceph:role: osd.1 2024-03-29T11:34:28.294 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:34:28.294 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:28.369 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:28.843 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-03-29T11:34:28.845 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-03-29T11:34:28.845 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:28.942 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:28.974 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:29.042 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:29.042 INFO:tasks.ceph:role: osd.2 2024-03-29T11:34:29.042 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:34:29.042 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:29.117 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:29.117 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:29.117 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:29.118 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:29.118 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:29.118 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:29.118 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:29.118 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:29.118 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:29.118 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:29.598 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-03-29T11:34:29.600 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-03-29T11:34:29.600 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:29.751 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:29.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:29.853 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:29.853 INFO:tasks.ceph:role: osd.3 2024-03-29T11:34:29.853 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:34:29.853 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:29.929 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:30.443 INFO:teuthology.orchestra.run.smithi145.stdout:Discarding blocks...Done. 2024-03-29T11:34:30.445 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-03-29T11:34:30.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:30.588 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:30.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:30.748 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:30.752+0000 7fd3cc437640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:34:30.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:30.752+0000 7fd3cc437640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:34:30.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:30.752+0000 7fd3cc437640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:30.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:30.752+0000 7fd3cc437640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:30.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:30.752+0000 7fd3cc437640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:34:30.749 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:30.753+0000 7fd3cc437640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:34:33.596 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:34:33.628 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:33.719 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:33.722+0000 7f5ea19ff640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:34:33.720 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:33.723+0000 7f5ea19ff640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:34:33.720 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:33.723+0000 7f5ea19ff640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:33.720 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:33.723+0000 7f5ea19ff640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:33.720 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:33.723+0000 7f5ea19ff640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:34:33.720 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:33.723+0000 7f5ea19ff640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:34:36.573 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:34:36.603 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:36.693 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:36.695+0000 7fdd40c11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:34:36.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:36.696+0000 7fdd40c11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:34:36.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:36.696+0000 7fdd40c11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:36.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:36.696+0000 7fdd40c11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:36.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:36.696+0000 7fdd40c11640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:34:36.694 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:36.696+0000 7fdd40c11640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:34:39.549 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:34:39.580 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:39.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:39.675+0000 7f6b58ec8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:34:39.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:39.675+0000 7f6b58ec8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:34:39.673 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:39.675+0000 7f6b58ec8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:39.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:39.675+0000 7f6b58ec8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:39.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:39.675+0000 7f6b58ec8640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:34:39.674 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:39.676+0000 7f6b58ec8640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:34:42.526 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:34:42.570 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:42.605 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:42.605 INFO:tasks.ceph:role: osd.4 2024-03-29T11:34:42.605 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:34:42.605 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:34:42.685 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:42.685 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:42.685 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:42.685 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:42.685 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:42.685 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:42.685 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:42.685 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:42.686 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:42.686 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:43.172 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-29T11:34:43.174 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-29T11:34:43.175 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:43.326 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:43.363 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:43.433 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:43.434 INFO:tasks.ceph:role: osd.5 2024-03-29T11:34:43.434 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:34:43.434 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:43.510 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:43.953 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-29T11:34:43.955 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-29T11:34:43.956 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:44.039 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:44.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:44.177 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:44.177 INFO:tasks.ceph:role: osd.6 2024-03-29T11:34:44.177 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:34:44.177 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:34:44.251 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:44.251 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:44.251 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:44.251 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:44.251 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:44.251 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:44.251 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:44.252 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:44.252 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:44.252 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:44.741 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-29T11:34:44.743 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-29T11:34:44.744 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:44.890 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:44.922 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:44.989 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:34:44.989 INFO:tasks.ceph:role: osd.7 2024-03-29T11:34:44.989 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:34:44.989 DEBUG:teuthology.orchestra.run.smithi203:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:34:45.064 INFO:teuthology.orchestra.run.smithi203.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:34:45.064 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:34:45.064 INFO:teuthology.orchestra.run.smithi203.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:34:45.064 INFO:teuthology.orchestra.run.smithi203.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:34:45.064 INFO:teuthology.orchestra.run.smithi203.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:34:45.064 INFO:teuthology.orchestra.run.smithi203.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:34:45.064 INFO:teuthology.orchestra.run.smithi203.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:34:45.064 INFO:teuthology.orchestra.run.smithi203.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:34:45.064 INFO:teuthology.orchestra.run.smithi203.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:34:45.065 INFO:teuthology.orchestra.run.smithi203.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:34:45.585 INFO:teuthology.orchestra.run.smithi203.stdout:Discarding blocks...Done. 2024-03-29T11:34:45.587 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi203.front.sepia.ceph.com -o noatime 2024-03-29T11:34:45.587 DEBUG:teuthology.orchestra.run.smithi203:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:45.716 DEBUG:teuthology.orchestra.run.smithi203:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:45.752 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:45.852 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:45.854+0000 7fb138619640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:34:45.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:45.855+0000 7fb138619640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:34:45.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:45.855+0000 7fb138619640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:45.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:45.855+0000 7fb138619640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:45.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:45.855+0000 7fb138619640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:34:45.853 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:45.855+0000 7fb138619640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:34:48.697 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:34:48.729 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:48.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:48.822+0000 7f7b05b28640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:34:48.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:48.823+0000 7f7b05b28640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:34:48.820 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:48.823+0000 7f7b05b28640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:48.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:48.823+0000 7f7b05b28640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:48.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:48.823+0000 7f7b05b28640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:34:48.821 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:48.823+0000 7f7b05b28640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:34:51.664 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:34:51.695 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:51.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:51.789+0000 7f99c840d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:34:51.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:51.789+0000 7f99c840d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:34:51.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:51.789+0000 7f99c840d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:51.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:51.789+0000 7f99c840d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:51.787 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:51.789+0000 7f99c840d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:34:51.788 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:51.790+0000 7f99c840d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:34:54.638 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:34:54.669 DEBUG:teuthology.orchestra.run.smithi203:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:54.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:54.760+0000 7f4f5a0d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:34:54.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:54.760+0000 7f4f5a0d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:34:54.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:54.760+0000 7f4f5a0d9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:54.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:54.760+0000 7f4f5a0d9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:54.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:54.760+0000 7f4f5a0d9640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:34:54.758 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:34:54.760+0000 7f4f5a0d9640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:34:57.598 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:34:57.627 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:34:57.627 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:57.627 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:34:57.659 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:57.659 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:34:57.731 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:57.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:34:57.800 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:57.800 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:34:57.870 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:57.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:34:57.939 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:57.939 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:34:57.969 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:57.969 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:34:58.038 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:58.038 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:34:58.109 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:58.110 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:34:58.183 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:58.183 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:34:58.254 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:34:58.254 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:34:58.277 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:34:58.277 DEBUG:teuthology.orchestra.run.smithi203:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:34:58.309 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:34:58.309 DEBUG:teuthology.orchestra.run.smithi145:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:58.321 DEBUG:teuthology.orchestra.run.smithi203:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.y] 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBDpwZm7o3gERAAX1CBbWHl8lI/rW5Y22wEuw== 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.0] 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBGpwZmUtDXLBAA5BfXQWny/BtnTFcVMP32bw== 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.1] 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBJpwZm6AQbKxAA5mVNwAbvGpvt0DlguSaZ1A== 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.2] 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBMpwZmx6+LKRAAMGhhzQ+Xr00gFhGNqC+IJQ== 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.3] 2024-03-29T11:34:58.354 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBPpwZmvvZUKBAArwr8iAonThXWhsOEUzgX8w== 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.x] 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBDpwZmN0gIFhAAGd2uI3wZfE6HU9TbTU6EWw== 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.4] 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBVpwZmzL0DMxAA05rWHKohJMJ4aXwwZmI1xw== 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.5] 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBYpwZmp78ZMRAAxZMbwq975mUmwS+P6eO0Hg== 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.6] 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBbpwZm/nscLxAA9DarnXloxT45WZgD9dUdKQ== 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.7] 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBepwZmydlcLRAA74vvdp9uIhTRrJCTuLOKYQ== 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout:[client.0] 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBDpwZm3X6wGBAAe35U/gA7JtBobV/BOs0y5A== 2024-03-29T11:34:58.355 INFO:teuthology.orchestra.run.smithi145.stdout:[client.1] 2024-03-29T11:34:58.356 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBDpwZmBJYZHBAA3lH2HgkV6oVW9UNUb4kG6w== 2024-03-29T11:34:58.376 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.y] 2024-03-29T11:34:58.376 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBDpwZm7o3gERAAX1CBbWHl8lI/rW5Y22wEuw== 2024-03-29T11:34:58.376 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.0] 2024-03-29T11:34:58.376 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBGpwZmUtDXLBAA5BfXQWny/BtnTFcVMP32bw== 2024-03-29T11:34:58.376 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.1] 2024-03-29T11:34:58.376 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBJpwZm6AQbKxAA5mVNwAbvGpvt0DlguSaZ1A== 2024-03-29T11:34:58.376 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.2] 2024-03-29T11:34:58.376 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBMpwZmx6+LKRAAMGhhzQ+Xr00gFhGNqC+IJQ== 2024-03-29T11:34:58.377 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.3] 2024-03-29T11:34:58.377 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBPpwZmvvZUKBAArwr8iAonThXWhsOEUzgX8w== 2024-03-29T11:34:58.377 INFO:teuthology.orchestra.run.smithi203.stdout:[mgr.x] 2024-03-29T11:34:58.377 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBDpwZmN0gIFhAAGd2uI3wZfE6HU9TbTU6EWw== 2024-03-29T11:34:58.377 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.4] 2024-03-29T11:34:58.377 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBVpwZmzL0DMxAA05rWHKohJMJ4aXwwZmI1xw== 2024-03-29T11:34:58.377 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.5] 2024-03-29T11:34:58.377 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBYpwZmp78ZMRAAxZMbwq975mUmwS+P6eO0Hg== 2024-03-29T11:34:58.378 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.6] 2024-03-29T11:34:58.378 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBbpwZm/nscLxAA9DarnXloxT45WZgD9dUdKQ== 2024-03-29T11:34:58.378 INFO:teuthology.orchestra.run.smithi203.stdout:[osd.7] 2024-03-29T11:34:58.378 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBepwZmydlcLRAA74vvdp9uIhTRrJCTuLOKYQ== 2024-03-29T11:34:58.378 INFO:teuthology.orchestra.run.smithi203.stdout:[client.0] 2024-03-29T11:34:58.378 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBDpwZm3X6wGBAAe35U/gA7JtBobV/BOs0y5A== 2024-03-29T11:34:58.378 INFO:teuthology.orchestra.run.smithi203.stdout:[client.1] 2024-03-29T11:34:58.378 INFO:teuthology.orchestra.run.smithi203.stdout: key = AQBDpwZmBJYZHBAA3lH2HgkV6oVW9UNUb4kG6w== 2024-03-29T11:34:58.380 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:58.404 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:58.474 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.481 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.534 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.537 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.593 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.650 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:58.706 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:34:58.758 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.761 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.813 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.815 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.870 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.922 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.925 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:34:58.977 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:58.979 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:59.033 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:59.035 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:34:59.090 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:34:59.090 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:59.117 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:59.215 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:34:59.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:59.308 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:59.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:34:59.432 DEBUG:teuthology.orchestra.run.smithi203:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:59.459 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:34:59.556 DEBUG:teuthology.orchestra.run.smithi203:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:34:59.583 DEBUG:teuthology.orchestra.run.smithi145:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:59.585 DEBUG:teuthology.orchestra.run.smithi203:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:34:59.638 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:34:59.638 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:34:59.639 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:34:59.641 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:34:59.641 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:34:59.641 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:34:59.643 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:34:59.643 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:34:59.643 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:34:59.681 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:34:59.681 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:34:59.681 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:34:59.681 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:34:59.683 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:34:59.683 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:34:59.683 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:34:59.685 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:34:59.685 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:34:59.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.789+0000 7f26157d4640 10 monclient: get_monmap_and_config 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.789+0000 7f26157d4640 10 monclient: build_initial_monmap 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.789+0000 7f26157d4640 10 monclient: monmap: 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 0 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:59.791149+0000 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:59.791149+0000 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:34:59.788 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:34:59.789 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:34:59.789 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:34:59.789 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:34:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.790+0000 7f26157d4640 1 Processor -- start 2024-03-29T11:34:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.790+0000 7f26157d4640 1 -- start start 2024-03-29T11:34:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.790+0000 7f26157d4640 10 monclient: init 2024-03-29T11:34:59.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 10 monclient: _reopen_session rank -1 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 10 monclient(hunting): picked mon.noname-b con 0x7f261009ac40 addr v1:172.21.15.145:6790/0 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 10 monclient(hunting): picked mon.noname-c con 0x7f261009c870 addr v1:172.21.15.203:6789/0 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 10 monclient(hunting): picked mon.noname-a con 0x7f26100b4360 addr v1:172.21.15.145:6789/0 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261009dec0 con 0x7f26100b4360 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261009da30 con 0x7f261009c870 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26100b39c0 con 0x7f261009ac40 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 10 monclient(hunting): _renew_subs 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f26157d4640 10 monclient(hunting): authenticate will time out at 889.099126s 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f260effd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f261009ac40 0x7f26100b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:42632/0 (socket says 172.21.15.145:42632) 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f260f7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f26100b4360 0x7f261014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:60238/0 (socket says 172.21.15.145:60238) 2024-03-29T11:34:59.791 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:34:59.791+0000 7f260f7fe640 1 -- 172.21.15.145:0/2062524608 learned_addr learned my addr 172.21.15.145:0/2062524608 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:02.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 10 monclient(hunting): tick 2024-03-29T11:35:02.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:35:02.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 1 monclient(hunting): continuing hunt 2024-03-29T11:35:02.789 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 1 -- 172.21.15.145:0/2062524608 >> v1:172.21.15.145:6790/0 conn(0x7f261009ac40 legacy=0x7f26100b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 1 -- 172.21.15.145:0/2062524608 >> v1:172.21.15.203:6789/0 conn(0x7f261009c870 legacy=0x7f261013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 1 -- 172.21.15.145:0/2062524608 >> v1:172.21.15.145:6789/0 conn(0x7f26100b4360 legacy=0x7f261014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f25fc0036e0 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f25fc007300 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f25fc00b020 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 1 -- 172.21.15.145:0/2062524608 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26100b5160 con 0x7f25fc0036e0 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 1 -- 172.21.15.145:0/2062524608 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc00fe30 con 0x7f25fc00b020 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 1 -- 172.21.15.145:0/2062524608 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25fc011070 con 0x7f25fc007300 2024-03-29T11:35:02.790 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:02.791+0000 7f260cff9640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:04.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874204008 0 0) 0x7f26100b5160 con 0x7f25fc0036e0 2024-03-29T11:35:04.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:35:04.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25f8003520 con 0x7f25fc0036e0 2024-03-29T11:35:04.956 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582106911 0 0) 0x7f25fc00fe30 con 0x7f25fc00b020 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26100b5160 con 0x7f25fc00b020 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893894265 0 0) 0x7f25fc011070 con 0x7f25fc007300 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.957+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25fc00fe30 con 0x7f25fc007300 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1170212139 0 0) 0x7f26100b5160 con 0x7f25fc00b020 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 >> v1:172.21.15.145:6790/0 conn(0x7f25fc007300 legacy=0x7f25fc009790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 >> v1:172.21.15.145:6789/0 conn(0x7f25fc0036e0 legacy=0x7f25fc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.957 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 1 monclient: found mon.noname-c 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25fc0122b0 con 0x7f25fc00b020 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2608001870 con 0x7f25fc00b020 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f26157d4640 5 monclient: authenticate success, global_id 4105 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f26157d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: dump: 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:04.958 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f26157d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2608001a40 con 0x7f25fc00b020 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 1 -- 172.21.15.145:0/2062524608 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2608001dd0 con 0x7f25fc00b020 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f26157d4640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f26157d4640 4 set_mon_vals no callback set 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.959 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.958+0000 7f260dffb640 10 monclient: dump: 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:04.960 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:04.964 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.959+0000 7f26157d4640 10 monclient: shutdown 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.959+0000 7f26157d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.959+0000 7f26157d4640 1 -- 172.21.15.145:0/2062524608 >> v1:172.21.15.203:6789/0 conn(0x7f25fc00b020 legacy=0x7f25fc00d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.959+0000 7f26157d4640 1 -- 172.21.15.145:0/2062524608 shutdown_connections 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.959+0000 7f26157d4640 1 -- 172.21.15.145:0/2062524608 >> 172.21.15.145:0/2062524608 conn(0x7f2610066b00 msgr2=0x7f2610066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.959+0000 7f26157d4640 1 -- 172.21.15.145:0/2062524608 shutdown_connections 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.959+0000 7f26157d4640 1 -- 172.21.15.145:0/2062524608 wait complete. 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.960+0000 7f26157d4640 10 monclient: build_initial_monmap 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.960+0000 7f26157d4640 10 monclient: monmap: 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 0 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:04.965 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:created 0.000000 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.960+0000 7f26157d4640 1 librados: starting msgr at 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.960+0000 7f26157d4640 1 librados: starting objecter 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.961+0000 7f26157d4640 1 Processor -- start 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.961+0000 7f26157d4640 1 -- start start 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.961+0000 7f26157d4640 1 librados: setting wanted keys 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.961+0000 7f26157d4640 1 librados: calling monclient init 2024-03-29T11:35:04.966 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.961+0000 7f26157d4640 10 monclient: init 2024-03-29T11:35:04.967 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:04.967 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:04.967 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 10 monclient(hunting): picked mon.noname-c con 0x7f25fc0036e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:04.967 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 10 monclient(hunting): picked mon.noname-a con 0x7f25fc007300 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 10 monclient(hunting): picked mon.noname-b con 0x7f25fc00b020 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26100ae280 con 0x7f25fc007300 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26100ae4b0 con 0x7f25fc00b020 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26100ad410 con 0x7f25fc0036e0 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f26157d4640 10 monclient(hunting): authenticate will time out at 894.270183s 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f260e7fc640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f25fc007300 0x7f26101d7550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36436/0 (socket says 172.21.15.145:36436) 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f260e7fc640 1 -- 172.21.15.145:0/1398797972 learned_addr learned my addr 172.21.15.145:0/1398797972 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 client.?.objecter ms_handle_connect 0x7f25fc007300 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 client.?.objecter ms_handle_connect 0x7f25fc0036e0 2024-03-29T11:35:04.969 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 client.?.objecter ms_handle_connect 0x7f25fc00b020 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129377151 0 0) 0x7f26100ae280 con 0x7f25fc007300 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.962+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25fc006a20 con 0x7f25fc007300 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4094433868 0 0) 0x7f25fc006a20 con 0x7f25fc007300 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26100ae280 con 0x7f25fc007300 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2604001930 con 0x7f25fc007300 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:04.970 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient(hunting): dump: 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1956433904 0 0) 0x7f26100ae280 con 0x7f25fc007300 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 >> v1:172.21.15.145:6790/0 conn(0x7f25fc0036e0 legacy=0x7f26100acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 >> v1:172.21.15.203:6789/0 conn(0x7f25fc00b020 legacy=0x7f2610138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:04.971 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 monclient: found mon.a 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26100b5840 con 0x7f25fc007300 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f26157d4640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f26157d4640 10 monclient: _renew_subs 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f26157d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f26157d4640 1 -- 172.21.15.145:0/1398797972 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26101d7c60 con 0x7f25fc007300 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f26157d4640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f26157d4640 10 monclient: _renew_subs 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f26157d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f26157d4640 1 -- 172.21.15.145:0/1398797972 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26100b81a0 con 0x7f25fc007300 2024-03-29T11:35:04.972 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f26157d4640 1 librados: init done 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26040035f0 con 0x7f25fc007300 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2604003a00 con 0x7f25fc007300 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 monclient: dump: 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:04.973 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f260d7fa640 4 set_mon_vals no callback set 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2604003fb0 con 0x7f25fc007300 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 76954437 0 0) 0x7f26040065f0 con 0x7f25fc007300 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 10 client.4110.objecter ms_dispatch 0x7f2610000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.963+0000 7f25f77fe640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.964+0000 7f26157d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.964+0000 7f26157d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.964+0000 7f26157d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.964+0000 7f26157d4640 1 -- 172.21.15.145:0/1398797972 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25f0005080 con 0x7f25fc007300 2024-03-29T11:35:04.974 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.967+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2604006830 con 0x7f25fc007300 2024-03-29T11:35:04.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.967+0000 7f25f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:04.975 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:04.967+0000 7f25f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:05.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.039+0000 7f26157d4640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:35:05.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.039+0000 7f26157d4640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:35:05.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.039+0000 7f26157d4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:05.037 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.039+0000 7f26157d4640 1 -- 172.21.15.145:0/1398797972 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f25f0005330 con 0x7f25fc007300 2024-03-29T11:35:05.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.041+0000 7f25f77fe640 1 -- 172.21.15.145:0/1398797972 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f2604005780 con 0x7f25fc007300 2024-03-29T11:35:05.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.041+0000 7f25f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:35:05.039 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.041+0000 7f25f77fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:35:05.039 INFO:teuthology.orchestra.run.smithi145.stderr:adjusted tunables profile to default 2024-03-29T11:35:05.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.041+0000 7f26157d4640 10 librados: watch_flush enter 2024-03-29T11:35:05.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.041+0000 7f26157d4640 10 librados: watch_flush exit 2024-03-29T11:35:05.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.041+0000 7f26157d4640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:35:05.040 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.041+0000 7f26157d4640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:35:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.042+0000 7f26157d4640 10 monclient: shutdown 2024-03-29T11:35:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.042+0000 7f26157d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.042+0000 7f26157d4640 1 -- 172.21.15.145:0/1398797972 >> v1:172.21.15.145:6789/0 conn(0x7f25fc007300 legacy=0x7f26101d7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.042+0000 7f26157d4640 1 -- 172.21.15.145:0/1398797972 shutdown_connections 2024-03-29T11:35:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.042+0000 7f26157d4640 1 -- 172.21.15.145:0/1398797972 >> 172.21.15.145:0/1398797972 conn(0x7f2610066b00 msgr2=0x7f2610066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.042+0000 7f26157d4640 1 -- 172.21.15.145:0/1398797972 shutdown_connections 2024-03-29T11:35:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.042+0000 7f26157d4640 1 -- 172.21.15.145:0/1398797972 wait complete. 2024-03-29T11:35:05.041 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:05.042+0000 7f26157d4640 1 librados: shutdown 2024-03-29T11:35:05.051 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:35:05.051 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:35:05.051 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:35:05.052 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:35:05.079 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:35:05.079 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:35:05.145 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:35:05.145 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:35:05.214 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-03-29T11:35:05.214 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:35:05.281 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:35:05.282 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:35:05.308 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:35:05.309 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:35:05.375 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:35:05.376 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:35:05.440 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:05.442+0000 7fa732632040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:35:05.442 DEBUG:teuthology.orchestra.run.smithi203:> set -ex 2024-03-29T11:35:05.442 DEBUG:teuthology.orchestra.run.smithi203:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:35:05.478 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:05.480+0000 7fe599e1e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:35:05.496 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:05.499+0000 7fa732632040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:35:05.509 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 3dc3165d-1d81-40d7-913b-378677dd8310 0 2024-03-29T11:35:05.532 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:05.534+0000 7fe599e1e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:35:05.649 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.651+0000 7f8f76e74640 10 monclient: get_monmap_and_config 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.651+0000 7f8f76e74640 10 monclient: build_initial_monmap 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.651+0000 7f8f76e74640 10 monclient: monmap: 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:35:05.652855+0000 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:35:05.652855+0000 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:05.650 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.652+0000 7f8f76e74640 1 Processor -- start 2024-03-29T11:35:05.659 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.652+0000 7f8f76e74640 1 -- start start 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.652+0000 7f8f76e74640 10 monclient: init 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 10 monclient(hunting): picked mon.noname-c con 0x7f8f7009ac40 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 10 monclient(hunting): picked mon.noname-a con 0x7f8f7009c870 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 10 monclient(hunting): picked mon.noname-b con 0x7f8f700b4360 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7009dec0 con 0x7f8f7009c870 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f7009da30 con 0x7f8f7009ac40 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f700b39c0 con 0x7f8f700b4360 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 10 monclient(hunting): authenticate will time out at 882.828157s 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f74c10640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f8f7009ac40 0x7f8f700b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45412/0 (socket says 172.21.15.203:45412) 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f74c10640 1 -- 172.21.15.203:0/2550003663 learned_addr learned my addr 172.21.15.203:0/2550003663 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:05.660 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1188623518 0 0) 0x7f8f7009da30 con 0x7f8f7009ac40 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f60003520 con 0x7f8f7009ac40 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557166614 0 0) 0x7f8f5c001110 con 0x7f8f7009c870 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f7009da30 con 0x7f8f7009c870 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929630587 0 0) 0x7f8f700b39c0 con 0x7f8f700b4360 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.661 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f5c001110 con 0x7f8f700b4360 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682761654 0 0) 0x7f8f60003520 con 0x7f8f7009ac40 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 >> v1:172.21.15.145:6790/0 conn(0x7f8f700b4360 legacy=0x7f8f7014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 >> v1:172.21.15.145:6789/0 conn(0x7f8f7009c870 legacy=0x7f8f7013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f700b5160 con 0x7f8f7009ac40 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:05.662 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8f64001690 con 0x7f8f7009ac40 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f6f7fe640 10 monclient: dump: 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.663 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.653+0000 7f8f76e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f64001870 con 0x7f8f7009ac40 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f6f7fe640 1 -- 172.21.15.203:0/2550003663 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8f64001c80 con 0x7f8f7009ac40 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f76e74640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f76e74640 4 set_mon_vals no callback set 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f6f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f6f7fe640 10 monclient: dump: 2024-03-29T11:35:05.664 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f76e74640 10 monclient: shutdown 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f76e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f76e74640 1 -- 172.21.15.203:0/2550003663 >> v1:172.21.15.203:6789/0 conn(0x7f8f7009ac40 legacy=0x7f8f700b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f76e74640 1 -- 172.21.15.203:0/2550003663 shutdown_connections 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f76e74640 1 -- 172.21.15.203:0/2550003663 >> 172.21.15.203:0/2550003663 conn(0x7f8f70066b00 msgr2=0x7f8f70066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:05.665 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f76e74640 1 -- 172.21.15.203:0/2550003663 shutdown_connections 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.654+0000 7f8f76e74640 1 -- 172.21.15.203:0/2550003663 wait complete. 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.655+0000 7f8f76e74640 10 monclient: build_initial_monmap 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.655+0000 7f8f76e74640 10 monclient: monmap: 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.666 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.655+0000 7f8f76e74640 1 librados: starting msgr at 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.655+0000 7f8f76e74640 1 librados: starting objecter 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.655+0000 7f8f76e74640 1 Processor -- start 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.655+0000 7f8f76e74640 1 -- start start 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.655+0000 7f8f76e74640 1 librados: setting wanted keys 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.655+0000 7f8f76e74640 1 librados: calling monclient init 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.655+0000 7f8f76e74640 10 monclient: init 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 10 monclient(hunting): picked mon.noname-a con 0x7f8f7009ac40 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 10 monclient(hunting): picked mon.noname-c con 0x7f8f70142e20 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 10 monclient(hunting): picked mon.noname-b con 0x7f8f700b0fd0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f701cb340 con 0x7f8f7009ac40 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f701cb080 con 0x7f8f700b0fd0 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f700b1a70 con 0x7f8f70142e20 2024-03-29T11:35:05.667 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f76e74640 10 monclient(hunting): authenticate will time out at 882.831211s 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f75411640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f8f700b0fd0 0x7f8f701e1a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45424/0 (socket says 172.21.15.203:45424) 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f75411640 1 -- 172.21.15.203:0/2293451088 learned_addr learned my addr 172.21.15.203:0/2293451088 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f8f700b0fd0 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f8f7009ac40 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 client.?.objecter ms_handle_connect 0x7f8f70142e20 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618588044 0 0) 0x7f8f701cb080 con 0x7f8f700b0fd0 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:35:05.668 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f54004eb0 con 0x7f8f700b0fd0 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491518990 0 0) 0x7f8f700b1a70 con 0x7f8f70142e20 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f701cb080 con 0x7f8f70142e20 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360707688 0 0) 0x7f8f701cb340 con 0x7f8f7009ac40 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f700b1a70 con 0x7f8f7009ac40 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3131699970 0 0) 0x7f8f701cb080 con 0x7f8f70142e20 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f701cb340 con 0x7f8f70142e20 2024-03-29T11:35:05.669 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8f5c0018e0 con 0x7f8f70142e20 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): dump: 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.670 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.656+0000 7f8f6d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1965588827 0 0) 0x7f8f54004eb0 con 0x7f8f700b0fd0 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f701cb080 con 0x7f8f700b0fd0 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374320339 0 0) 0x7f8f700b1a70 con 0x7f8f7009ac40 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f54004eb0 con 0x7f8f7009ac40 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8f68001bf0 con 0x7f8f700b0fd0 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient(hunting): dump: 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.671 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8f640019e0 con 0x7f8f7009ac40 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient(hunting): dump: 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.672 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 160969335 0 0) 0x7f8f701cb340 con 0x7f8f70142e20 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 >> v1:172.21.15.203:6789/0 conn(0x7f8f700b0fd0 legacy=0x7f8f701e1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 >> v1:172.21.15.145:6789/0 conn(0x7f8f7009ac40 legacy=0x7f8f701ca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 monclient: found mon.c 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f701dabc0 con 0x7f8f70142e20 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:05.673 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f76e74640 5 monclient: authenticate success, global_id 4118 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f76e74640 10 monclient: _renew_subs 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f76e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f76e74640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f701e3180 con 0x7f8f70142e20 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f76e74640 10 client.4118.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f76e74640 10 monclient: _renew_subs 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f76e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f76e74640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f701e34f0 con 0x7f8f70142e20 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f76e74640 1 librados: init done 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f5c001d70 con 0x7f8f70142e20 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8f5c003790 con 0x7f8f70142e20 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:05.674 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6effd640 4 set_mon_vals no callback set 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 monclient: dump: 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8f5c003d00 con 0x7f8f70142e20 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 1697807571 0 0) 0x7f8f5c005290 con 0x7f8f70142e20 2024-03-29T11:35:05.675 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 10 client.4118.objecter ms_dispatch 0x7f8f70000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 3 client.4118.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 3 client.4118.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.657+0000 7f8f6d7fa640 20 client.4118.objecter dump_active .. 0 homeless 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.658+0000 7f8f76e74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.658+0000 7f8f76e74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.658+0000 7f8f76e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.658+0000 7f8f76e74640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f50005080 con 0x7f8f70142e20 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.660+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8f5c004190 con 0x7f8f70142e20 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.660+0000 7f8f6d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:05.676 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.660+0000 7f8f6d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:05.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.732+0000 7f8f76e74640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3dc3165d-1d81-40d7-913b-378677dd8310", "id": 0}] 2024-03-29T11:35:05.730 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.732+0000 7f8f76e74640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3dc3165d-1d81-40d7-913b-378677dd8310", "id": 0}] 2024-03-29T11:35:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.732+0000 7f8f76e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:05.731 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.732+0000 7f8f76e74640 1 -- 172.21.15.203:0/2293451088 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd new", "uuid": "3dc3165d-1d81-40d7-913b-378677dd8310", "id": 0} v 0) v1 -- 0x7f8f500058b0 con 0x7f8f70142e20 2024-03-29T11:35:05.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.735+0000 7f8f6d7fa640 1 -- 172.21.15.203:0/2293451088 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3dc3165d-1d81-40d7-913b-378677dd8310", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3642766435 0 732707515) 0x7f8f5c004480 con 0x7f8f70142e20 2024-03-29T11:35:05.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.735+0000 7f8f6d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3dc3165d-1d81-40d7-913b-378677dd8310", "id": 0}] 2024-03-29T11:35:05.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.735+0000 7f8f6d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:05.733 INFO:teuthology.orchestra.run.smithi203.stdout:0 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 10 librados: watch_flush enter 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 10 librados: watch_flush exit 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 20 client.4118.objecter shutdown clearing up homeless session... 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 10 client.4118.objecter successfully canceled tick 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 10 monclient: shutdown 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 1 -- 172.21.15.203:0/2293451088 >> v1:172.21.15.145:6790/0 conn(0x7f8f70142e20 legacy=0x7f8f701da4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 1 -- 172.21.15.203:0/2293451088 shutdown_connections 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 1 -- 172.21.15.203:0/2293451088 >> 172.21.15.203:0/2293451088 conn(0x7f8f70066b00 msgr2=0x7f8f70066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 1 -- 172.21.15.203:0/2293451088 shutdown_connections 2024-03-29T11:35:05.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 1 -- 172.21.15.203:0/2293451088 wait complete. 2024-03-29T11:35:05.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.736+0000 7f8f76e74640 1 librados: shutdown 2024-03-29T11:35:05.747 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 35f88c45-b7c4-406d-b24a-9b3f432b80ad 1 2024-03-29T11:35:05.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.849+0000 7f23ceac1640 10 monclient: get_monmap_and_config 2024-03-29T11:35:05.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.849+0000 7f23ceac1640 10 monclient: build_initial_monmap 2024-03-29T11:35:05.847 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.849+0000 7f23ceac1640 10 monclient: monmap: 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:35:05.850764+0000 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:35:05.850764+0000 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:05.848 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.850+0000 7f23ceac1640 1 Processor -- start 2024-03-29T11:35:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.850+0000 7f23ceac1640 1 -- start start 2024-03-29T11:35:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.850+0000 7f23ceac1640 10 monclient: init 2024-03-29T11:35:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 10 monclient(hunting): picked mon.noname-b con 0x7f23c809ac40 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 10 monclient(hunting): picked mon.noname-c con 0x7f23c809c870 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:05.857 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 10 monclient(hunting): picked mon.noname-a con 0x7f23c80b4360 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c809dec0 con 0x7f23c80b4360 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c809da30 con 0x7f23c809c870 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80b39c0 con 0x7f23c809ac40 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 10 monclient(hunting): authenticate will time out at 883.026053s 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c7fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f23c809c870 0x7f23c813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45434/0 (socket says 172.21.15.203:45434) 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c7fff640 1 -- 172.21.15.203:0/3118754361 learned_addr learned my addr 172.21.15.203:0/3118754361 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023284812 0 0) 0x7f23c809da30 con 0x7f23c809c870 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b8003520 con 0x7f23c809c870 2024-03-29T11:35:05.858 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602353601 0 0) 0x7f23c0001490 con 0x7f23c80b4360 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c809da30 con 0x7f23c80b4360 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392067320 0 0) 0x7f23c80b39c0 con 0x7f23c809ac40 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c0001490 con 0x7f23c809ac40 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3515062593 0 0) 0x7f23b8003520 con 0x7f23c809c870 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 >> v1:172.21.15.145:6790/0 conn(0x7f23c809ac40 legacy=0x7f23c80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.859 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 >> v1:172.21.15.145:6789/0 conn(0x7f23c80b4360 legacy=0x7f23c814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 monclient: found mon.noname-c 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80b5160 con 0x7f23c809c870 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f23b4001690 con 0x7f23c809c870 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: dump: 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.860 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23b4001870 con 0x7f23c809c870 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 1 -- 172.21.15.203:0/3118754361 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f23b4001c80 con 0x7f23c809c870 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23ceac1640 4 set_mon_vals no callback set 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.861 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.851+0000 7f23c77fe640 10 monclient: dump: 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.852+0000 7f23ceac1640 10 monclient: shutdown 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.852+0000 7f23ceac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.852+0000 7f23ceac1640 1 -- 172.21.15.203:0/3118754361 >> v1:172.21.15.203:6789/0 conn(0x7f23c809c870 legacy=0x7f23c813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.862 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.852+0000 7f23ceac1640 1 -- 172.21.15.203:0/3118754361 shutdown_connections 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.852+0000 7f23ceac1640 1 -- 172.21.15.203:0/3118754361 >> 172.21.15.203:0/3118754361 conn(0x7f23c8066b00 msgr2=0x7f23c8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.852+0000 7f23ceac1640 1 -- 172.21.15.203:0/3118754361 shutdown_connections 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.852+0000 7f23ceac1640 1 -- 172.21.15.203:0/3118754361 wait complete. 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.853+0000 7f23ceac1640 10 monclient: build_initial_monmap 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.853+0000 7f23ceac1640 10 monclient: monmap: 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:05.863 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.853+0000 7f23ceac1640 1 librados: starting msgr at 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.853+0000 7f23ceac1640 1 librados: starting objecter 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.853+0000 7f23ceac1640 1 Processor -- start 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.853+0000 7f23ceac1640 1 -- start start 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.853+0000 7f23ceac1640 1 librados: setting wanted keys 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.853+0000 7f23ceac1640 1 librados: calling monclient init 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.853+0000 7f23ceac1640 10 monclient: init 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 10 monclient(hunting): picked mon.noname-c con 0x7f23c809ac40 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 10 monclient(hunting): picked mon.noname-a con 0x7f23c80af410 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 10 monclient(hunting): picked mon.noname-b con 0x7f23c81dafd0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80b1590 con 0x7f23c80af410 2024-03-29T11:35:05.864 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c81da3c0 con 0x7f23c81dafd0 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c80b1800 con 0x7f23c809ac40 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23ceac1640 10 monclient(hunting): authenticate will time out at 883.029044s 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23cd05e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f23c81dafd0 0x7f23c8139040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45450/0 (socket says 172.21.15.203:45450) 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23cd05e640 1 -- 172.21.15.203:0/503620890 learned_addr learned my addr 172.21.15.203:0/503620890 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 client.?.objecter ms_handle_connect 0x7f23c81dafd0 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 client.?.objecter ms_handle_connect 0x7f23c809ac40 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 client.?.objecter ms_handle_connect 0x7f23c80af410 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277958994 0 0) 0x7f23c81da3c0 con 0x7f23c81dafd0 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.865 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23ac004eb0 con 0x7f23c81dafd0 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144486828 0 0) 0x7f23c80b1590 con 0x7f23c80af410 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c81da3c0 con 0x7f23c80af410 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3340939189 0 0) 0x7f23c80b1800 con 0x7f23c809ac40 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:35:05.866 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c80b1590 con 0x7f23c809ac40 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4119075952 0 0) 0x7f23ac004eb0 con 0x7f23c81dafd0 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c80b1800 con 0x7f23c81dafd0 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f23c0001bf0 con 0x7f23c81dafd0 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): dump: 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.867 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4237960139 0 0) 0x7f23c81da3c0 con 0x7f23c80af410 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23ac004eb0 con 0x7f23c80af410 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f23b4001b80 con 0x7f23c80af410 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): dump: 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.868 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 661986516 0 0) 0x7f23c80b1590 con 0x7f23c809ac40 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23c81da3c0 con 0x7f23c809ac40 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f23bc001830 con 0x7f23c809ac40 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 10 monclient(hunting): dump: 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.869 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2796669576 0 0) 0x7f23c80b1800 con 0x7f23c81dafd0 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.854+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 >> v1:172.21.15.145:6790/0 conn(0x7f23c809ac40 legacy=0x7f23c80afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 >> v1:172.21.15.145:6789/0 conn(0x7f23c80af410 legacy=0x7f23c81da8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 1 monclient: found mon.b 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c80acd00 con 0x7f23c81dafd0 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:05.870 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 10 monclient: _renew_subs 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c8139750 con 0x7f23c81dafd0 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 10 monclient: _renew_subs 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c81e3770 con 0x7f23c81dafd0 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 1 librados: init done 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23c0003990 con 0x7f23c81dafd0 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f23c0003ca0 con 0x7f23c81dafd0 2024-03-29T11:35:05.871 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 10 monclient: dump: 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c6ffd640 4 set_mon_vals no callback set 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f23c0004210 con 0x7f23c81dafd0 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 4088194720 0 0) 0x7f23c0006890 con 0x7f23c81dafd0 2024-03-29T11:35:05.872 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 10 client.4117.objecter ms_dispatch 0x7f23c8000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23c57fa640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.855+0000 7f23ceac1640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a8005080 con 0x7f23c81dafd0 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.859+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f23c00046a0 con 0x7f23c81dafd0 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.859+0000 7f23c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:05.873 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.859+0000 7f23c57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:05.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.932+0000 7f23ceac1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "35f88c45-b7c4-406d-b24a-9b3f432b80ad", "id": 1}] 2024-03-29T11:35:05.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.932+0000 7f23ceac1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "35f88c45-b7c4-406d-b24a-9b3f432b80ad", "id": 1}] 2024-03-29T11:35:05.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.932+0000 7f23ceac1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:05.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.932+0000 7f23ceac1640 1 -- 172.21.15.203:0/503620890 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "35f88c45-b7c4-406d-b24a-9b3f432b80ad", "id": 1} v 0) v1 -- 0x7f23a80058b0 con 0x7f23c81dafd0 2024-03-29T11:35:05.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.940+0000 7f23c57fa640 1 -- 172.21.15.203:0/503620890 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "35f88c45-b7c4-406d-b24a-9b3f432b80ad", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3831288650 0 940483276) 0x7f23c0006aa0 con 0x7f23c81dafd0 2024-03-29T11:35:05.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.940+0000 7f23c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "35f88c45-b7c4-406d-b24a-9b3f432b80ad", "id": 1}] 2024-03-29T11:35:05.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.940+0000 7f23c57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:05.939 INFO:teuthology.orchestra.run.smithi203.stdout:1 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.941+0000 7f23ceac1640 10 librados: watch_flush enter 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.941+0000 7f23ceac1640 10 librados: watch_flush exit 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.941+0000 7f23ceac1640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.941+0000 7f23ceac1640 10 client.4117.objecter successfully canceled tick 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.941+0000 7f23ceac1640 10 monclient: shutdown 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.941+0000 7f23ceac1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.941+0000 7f23ceac1640 1 -- 172.21.15.203:0/503620890 >> v1:172.21.15.203:6789/0 conn(0x7f23c81dafd0 legacy=0x7f23c8139040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.941+0000 7f23ceac1640 1 -- 172.21.15.203:0/503620890 shutdown_connections 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.941+0000 7f23ceac1640 1 -- 172.21.15.203:0/503620890 >> 172.21.15.203:0/503620890 conn(0x7f23c8066b00 msgr2=0x7f23c813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:05.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.942+0000 7f23ceac1640 1 -- 172.21.15.203:0/503620890 shutdown_connections 2024-03-29T11:35:05.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.942+0000 7f23ceac1640 1 -- 172.21.15.203:0/503620890 wait complete. 2024-03-29T11:35:05.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:05.942+0000 7f23ceac1640 1 librados: shutdown 2024-03-29T11:35:05.950 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new f33f2af2-bb94-43d0-96d0-6b99d0a736fa 2 2024-03-29T11:35:06.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.052+0000 7fc1a0eff640 10 monclient: get_monmap_and_config 2024-03-29T11:35:06.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.052+0000 7fc1a0eff640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.050 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.052+0000 7fc1a0eff640 10 monclient: monmap: 2024-03-29T11:35:06.050 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.050 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:35:06.053277+0000 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:35:06.053277+0000 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.052+0000 7fc1a0eff640 1 Processor -- start 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.052+0000 7fc1a0eff640 1 -- start start 2024-03-29T11:35:06.051 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.052+0000 7fc1a0eff640 10 monclient: init 2024-03-29T11:35:06.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:06.059 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 10 monclient(hunting): picked mon.noname-b con 0x7fc19c09ac40 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 10 monclient(hunting): picked mon.noname-c con 0x7fc19c09c870 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 10 monclient(hunting): picked mon.noname-a con 0x7fc19c0b4360 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c09dec0 con 0x7fc19c0b4360 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c09da30 con 0x7fc19c09c870 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0b39c0 con 0x7fc19c09ac40 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc1a0eff640 10 monclient(hunting): authenticate will time out at 883.228536s 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc199d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc19c09c870 0x7fc19c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45452/0 (socket says 172.21.15.203:45452) 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.053+0000 7fc199d9b640 1 -- 172.21.15.203:0/2851484403 learned_addr learned my addr 172.21.15.203:0/2851484403 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542698411 0 0) 0x7fc19c09da30 con 0x7fc19c09c870 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.060 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc184003520 con 0x7fc19c09c870 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154039144 0 0) 0x7fc19c0b39c0 con 0x7fc19c09ac40 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c09da30 con 0x7fc19c09ac40 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273033586 0 0) 0x7fc194001490 con 0x7fc19c0b4360 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c0b39c0 con 0x7fc19c0b4360 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475122359 0 0) 0x7fc19c09da30 con 0x7fc19c09ac40 2024-03-29T11:35:06.061 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 >> v1:172.21.15.203:6789/0 conn(0x7fc19c09c870 legacy=0x7fc19c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 >> v1:172.21.15.145:6789/0 conn(0x7fc19c0b4360 legacy=0x7fc19c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 monclient: found mon.noname-b 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0b5160 con 0x7fc19c09ac40 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc190001670 con 0x7fc19c09ac40 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc1a0eff640 5 monclient: authenticate success, global_id 4127 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc1a0eff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:06.062 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: dump: 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc1a0eff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc190001850 con 0x7fc19c09ac40 2024-03-29T11:35:06.063 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 1 -- 172.21.15.203:0/2851484403 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc190001ca0 con 0x7fc19c09ac40 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc19959a640 10 monclient: dump: 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.064 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc1a0eff640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.054+0000 7fc1a0eff640 4 set_mon_vals no callback set 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.055+0000 7fc1a0eff640 10 monclient: shutdown 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.055+0000 7fc1a0eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.055+0000 7fc1a0eff640 1 -- 172.21.15.203:0/2851484403 >> v1:172.21.15.145:6790/0 conn(0x7fc19c09ac40 legacy=0x7fc19c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.055+0000 7fc1a0eff640 1 -- 172.21.15.203:0/2851484403 shutdown_connections 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.055+0000 7fc1a0eff640 1 -- 172.21.15.203:0/2851484403 >> 172.21.15.203:0/2851484403 conn(0x7fc19c066b00 msgr2=0x7fc19c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.055+0000 7fc1a0eff640 1 -- 172.21.15.203:0/2851484403 shutdown_connections 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.055+0000 7fc1a0eff640 1 -- 172.21.15.203:0/2851484403 wait complete. 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.056+0000 7fc1a0eff640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.056+0000 7fc1a0eff640 10 monclient: monmap: 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-29T11:35:06.065 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.057+0000 7fc1a0eff640 1 librados: starting msgr at 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.057+0000 7fc1a0eff640 1 librados: starting objecter 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.057+0000 7fc1a0eff640 1 Processor -- start 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.057+0000 7fc1a0eff640 1 -- start start 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.057+0000 7fc1a0eff640 1 librados: setting wanted keys 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.057+0000 7fc1a0eff640 1 librados: calling monclient init 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.057+0000 7fc1a0eff640 10 monclient: init 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.066 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 10 monclient(hunting): picked mon.noname-c con 0x7fc19c09ac40 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 10 monclient(hunting): picked mon.noname-a con 0x7fc19c0afb60 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 10 monclient(hunting): picked mon.noname-b con 0x7fc19c1df740 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c1dafb0 con 0x7fc19c0afb60 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0b1a50 con 0x7fc19c1df740 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c13ffa0 con 0x7fc19c09ac40 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 10 monclient(hunting): authenticate will time out at 883.232997s 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc199d9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fc19c0afb60 0x7fc19c0affc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.203:52600/0 (socket says 172.21.15.203:52600) 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc199d9b640 1 -- 172.21.15.203:0/1709635378 learned_addr learned my addr 172.21.15.203:0/1709635378 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc19ad9d640 1 --1- 172.21.15.203:0/1709635378 >> v1:172.21.15.203:6789/0 conn(0x7fc19c1df740 0x7fc19c1e1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45460/0 (socket says 172.21.15.203:45460) 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 client.?.objecter ms_handle_connect 0x7fc19c1df740 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 client.?.objecter ms_handle_connect 0x7fc19c0afb60 2024-03-29T11:35:06.067 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 client.?.objecter ms_handle_connect 0x7fc19c09ac40 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255086922 0 0) 0x7fc19c0b1a50 con 0x7fc19c1df740 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc17c004eb0 con 0x7fc19c1df740 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016760000 0 0) 0x7fc19c1dafb0 con 0x7fc19c0afb60 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c0b1a50 con 0x7fc19c0afb60 2024-03-29T11:35:06.068 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826735929 0 0) 0x7fc19c13ffa0 con 0x7fc19c09ac40 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c1dafb0 con 0x7fc19c09ac40 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2990758113 0 0) 0x7fc17c004eb0 con 0x7fc19c1df740 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc19c13ffa0 con 0x7fc19c1df740 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2698736602 0 0) 0x7fc19c1dafb0 con 0x7fc19c09ac40 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc17c004eb0 con 0x7fc19c09ac40 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1136553867 0 0) 0x7fc19c0b1a50 con 0x7fc19c0afb60 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc19c1dafb0 con 0x7fc19c0afb60 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc194001b50 con 0x7fc19c1df740 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.069 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): dump: 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc190001b60 con 0x7fc19c09ac40 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.070 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): dump: 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc18c0011b0 con 0x7fc19c0afb60 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.071 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient(hunting): dump: 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 57738026 0 0) 0x7fc17c004eb0 con 0x7fc19c09ac40 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 >> v1:172.21.15.203:6789/0 conn(0x7fc19c1df740 legacy=0x7fc19c1e1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 >> v1:172.21.15.145:6789/0 conn(0x7fc19c0afb60 legacy=0x7fc19c0affc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.072 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 monclient: found mon.c 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c140130 con 0x7fc19c09ac40 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc18affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.058+0000 7fc1a0eff640 5 monclient: authenticate success, global_id 4130 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc1a0eff640 10 monclient: _renew_subs 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc1a0eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc1a0eff640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c0b06d0 con 0x7fc19c09ac40 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc1a0eff640 10 client.4130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc1a0eff640 10 monclient: _renew_subs 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc1a0eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:06.073 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc1a0eff640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c1e3250 con 0x7fc19c09ac40 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc1a0eff640 1 librados: init done 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc190003960 con 0x7fc19c09ac40 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc190001000 con 0x7fc19c09ac40 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc198d99640 4 set_mon_vals no callback set 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 10 monclient: dump: 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.074 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc190004000 con 0x7fc19c09ac40 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 737581542 0 0) 0x7fc190005510 con 0x7fc19c09ac40 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 10 client.4130.objecter ms_dispatch 0x7fc19c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 3 client.4130.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 3 client.4130.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.059+0000 7fc18affd640 20 client.4130.objecter dump_active .. 0 homeless 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.060+0000 7fc1a0eff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.060+0000 7fc1a0eff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.060+0000 7fc1a0eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.060+0000 7fc1a0eff640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc180005080 con 0x7fc19c09ac40 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.062+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc190005150 con 0x7fc19c09ac40 2024-03-29T11:35:06.075 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.062+0000 7fc18affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.076 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.062+0000 7fc18affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:06.076 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.071+0000 7fa732632040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:35:06.076 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.078+0000 7fe599e1e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:35:06.124 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.127+0000 7fa732632040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:35:06.129 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.131+0000 7fe599e1e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:35:06.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.134+0000 7fc1a0eff640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f33f2af2-bb94-43d0-96d0-6b99d0a736fa", "id": 2}] 2024-03-29T11:35:06.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.134+0000 7fc1a0eff640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f33f2af2-bb94-43d0-96d0-6b99d0a736fa", "id": 2}] 2024-03-29T11:35:06.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.134+0000 7fc1a0eff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:06.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.134+0000 7fc1a0eff640 1 -- 172.21.15.203:0/1709635378 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd new", "uuid": "f33f2af2-bb94-43d0-96d0-6b99d0a736fa", "id": 2} v 0) v1 -- 0x7fc1800058b0 con 0x7fc19c09ac40 2024-03-29T11:35:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.137+0000 7fc18affd640 1 -- 172.21.15.203:0/1709635378 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f33f2af2-bb94-43d0-96d0-6b99d0a736fa", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3771468714 0 216599125) 0x7fc190005750 con 0x7fc19c09ac40 2024-03-29T11:35:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.137+0000 7fc18affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f33f2af2-bb94-43d0-96d0-6b99d0a736fa", "id": 2}] 2024-03-29T11:35:06.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.137+0000 7fc18affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:06.135 INFO:teuthology.orchestra.run.smithi203.stdout:2 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 10 librados: watch_flush enter 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 10 librados: watch_flush exit 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 20 client.4130.objecter shutdown clearing up homeless session... 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 10 client.4130.objecter successfully canceled tick 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 10 monclient: shutdown 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 1 -- 172.21.15.203:0/1709635378 >> v1:172.21.15.145:6790/0 conn(0x7fc19c09ac40 legacy=0x7fc19c1da410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 1 -- 172.21.15.203:0/1709635378 shutdown_connections 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 1 -- 172.21.15.203:0/1709635378 >> 172.21.15.203:0/1709635378 conn(0x7fc19c066b00 msgr2=0x7fc19c1cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.138+0000 7fc1a0eff640 1 -- 172.21.15.203:0/1709635378 shutdown_connections 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.139+0000 7fc1a0eff640 1 -- 172.21.15.203:0/1709635378 wait complete. 2024-03-29T11:35:06.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.139+0000 7fc1a0eff640 1 librados: shutdown 2024-03-29T11:35:06.147 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 42e5db63-c492-433f-810c-2e58ea1368cf 3 2024-03-29T11:35:06.173 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.175+0000 7fa732632040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:35:06.174 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.176+0000 7fe599e1e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:35:06.227 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.229+0000 7fe599e1e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:35:06.229 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.231+0000 7fa732632040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:35:06.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.248+0000 7f0469f72640 10 monclient: get_monmap_and_config 2024-03-29T11:35:06.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.248+0000 7f0469f72640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.246 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.248+0000 7f0469f72640 10 monclient: monmap: 2024-03-29T11:35:06.246 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.246 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.246 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:35:06.249611+0000 2024-03-29T11:35:06.246 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:35:06.249611+0000 2024-03-29T11:35:06.246 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.247 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.247 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.247 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:06.247 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:06.247 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.249+0000 7f0469f72640 1 Processor -- start 2024-03-29T11:35:06.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.249+0000 7f0469f72640 1 -- start start 2024-03-29T11:35:06.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.249+0000 7f0469f72640 10 monclient: init 2024-03-29T11:35:06.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:06.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 10 monclient(hunting): picked mon.noname-c con 0x7f046409ac40 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 10 monclient(hunting): picked mon.noname-b con 0x7f046409c870 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.256 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 10 monclient(hunting): picked mon.noname-a con 0x7f04640b4360 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046409dec0 con 0x7f04640b4360 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f046409da30 con 0x7f046409ac40 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640b39c0 con 0x7f046409c870 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 10 monclient(hunting): authenticate will time out at 883.425059s 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f04637fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f046409ac40 0x7f04640b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45464/0 (socket says 172.21.15.203:45464) 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f04637fe640 1 -- 172.21.15.203:0/2790659003 learned_addr learned my addr 172.21.15.203:0/2790659003 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890889462 0 0) 0x7f046409da30 con 0x7f046409ac40 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0450003520 con 0x7f046409ac40 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2548027385 0 0) 0x7f045c001490 con 0x7f04640b4360 2024-03-29T11:35:06.257 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f046409da30 con 0x7f04640b4360 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959455276 0 0) 0x7f04640b39c0 con 0x7f046409c870 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045c001490 con 0x7f046409c870 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4005469453 0 0) 0x7f0450003520 con 0x7f046409ac40 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 >> v1:172.21.15.145:6790/0 conn(0x7f046409c870 legacy=0x7f046413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 >> v1:172.21.15.145:6789/0 conn(0x7f04640b4360 legacy=0x7f046414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.258 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 monclient: found mon.noname-c 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04640b5160 con 0x7f046409ac40 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0454001690 con 0x7f046409ac40 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0462ffd640 10 monclient: dump: 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.259 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.250+0000 7f0469f72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0454001870 con 0x7f046409ac40 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0462ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0462ffd640 1 -- 172.21.15.203:0/2790659003 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0454001c80 con 0x7f046409ac40 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.260 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0462ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0462ffd640 10 monclient: dump: 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0469f72640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0469f72640 4 set_mon_vals no callback set 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0469f72640 10 monclient: shutdown 2024-03-29T11:35:06.261 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0469f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0469f72640 1 -- 172.21.15.203:0/2790659003 >> v1:172.21.15.203:6789/0 conn(0x7f046409ac40 legacy=0x7f04640b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0469f72640 1 -- 172.21.15.203:0/2790659003 shutdown_connections 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0469f72640 1 -- 172.21.15.203:0/2790659003 >> 172.21.15.203:0/2790659003 conn(0x7f0464066b00 msgr2=0x7f0464066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.251+0000 7f0469f72640 1 -- 172.21.15.203:0/2790659003 shutdown_connections 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.252+0000 7f0469f72640 1 -- 172.21.15.203:0/2790659003 wait complete. 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.252+0000 7f0469f72640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.252+0000 7f0469f72640 10 monclient: monmap: 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.262 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.253+0000 7f0469f72640 1 librados: starting msgr at 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.253+0000 7f0469f72640 1 librados: starting objecter 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.253+0000 7f0469f72640 1 Processor -- start 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.253+0000 7f0469f72640 1 -- start start 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.253+0000 7f0469f72640 1 librados: setting wanted keys 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.253+0000 7f0469f72640 1 librados: calling monclient init 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.253+0000 7f0469f72640 10 monclient: init 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 10 monclient(hunting): picked mon.noname-b con 0x7f046409ac40 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 10 monclient(hunting): picked mon.noname-a con 0x7f04640b1860 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 10 monclient(hunting): picked mon.noname-c con 0x7f046413b5b0 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04641d7660 con 0x7f04640b1860 2024-03-29T11:35:06.263 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04640ade20 con 0x7f046409ac40 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04641406b0 con 0x7f046413b5b0 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0469f72640 10 monclient(hunting): authenticate will time out at 883.429296s 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f04637fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f046409ac40 0x7f046413ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45474/0 (socket says 172.21.15.203:45474) 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f04637fe640 1 -- 172.21.15.203:0/417689025 learned_addr learned my addr 172.21.15.203:0/417689025 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f045bfff640 1 --1- 172.21.15.203:0/417689025 >> v1:172.21.15.145:6789/0 conn(0x7f04640b1860 0x7f046413aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.203:52624/0 (socket says 172.21.15.203:52624) 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 client.?.objecter ms_handle_connect 0x7f046409ac40 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 client.?.objecter ms_handle_connect 0x7f04640b1860 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 client.?.objecter ms_handle_connect 0x7f046413b5b0 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383862021 0 0) 0x7f04640ade20 con 0x7f046409ac40 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.264 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0444004eb0 con 0x7f046409ac40 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460706482 0 0) 0x7f04641d7660 con 0x7f04640b1860 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04640ade20 con 0x7f04640b1860 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806945236 0 0) 0x7f04641406b0 con 0x7f046413b5b0 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:35:06.265 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.254+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04641d7660 con 0x7f046413b5b0 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1233935764 0 0) 0x7f0444004eb0 con 0x7f046409ac40 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04641406b0 con 0x7f046409ac40 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0454001bc0 con 0x7f046409ac40 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.266 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 782383001 0 0) 0x7f04640ade20 con 0x7f04640b1860 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0444004eb0 con 0x7f04640b1860 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1537848269 0 0) 0x7f04641d7660 con 0x7f046413b5b0 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04640ade20 con 0x7f046413b5b0 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f044c001870 con 0x7f04640b1860 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.267 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f045c001a40 con 0x7f046413b5b0 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.268 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2388485418 0 0) 0x7f04641406b0 con 0x7f046409ac40 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 >> v1:172.21.15.145:6790/0 conn(0x7f046413b5b0 legacy=0x7f04641e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 >> v1:172.21.15.145:6789/0 conn(0x7f04640b1860 legacy=0x7f046413aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 monclient: found mon.b 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04641cccd0 con 0x7f046409ac40 2024-03-29T11:35:06.269 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0469f72640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0469f72640 10 monclient: _renew_subs 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0469f72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0469f72640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04641cce60 con 0x7f046409ac40 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0469f72640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0469f72640 10 monclient: _renew_subs 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0469f72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0469f72640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04641d7190 con 0x7f046409ac40 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0469f72640 1 librados: init done 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0454004bf0 con 0x7f046409ac40 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.270 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0454004f00 con 0x7f046409ac40 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f04627fc640 4 set_mon_vals no callback set 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 monclient: dump: 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.271 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f04540053f0 con 0x7f046409ac40 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 87406767 0 0) 0x7f0454006940 con 0x7f046409ac40 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 10 client.4129.objecter ms_dispatch 0x7f0464000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.255+0000 7f0460ff9640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.256+0000 7f0469f72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.256+0000 7f0469f72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.256+0000 7f0469f72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.256+0000 7f0469f72640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0448005080 con 0x7f046409ac40 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.258+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0454006560 con 0x7f046409ac40 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.258+0000 7f0460ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.272 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.258+0000 7f0460ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:06.273 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.272+0000 7fe599e1e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:35:06.275 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.278+0000 7fa732632040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:35:06.328 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.331+0000 7f0469f72640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "42e5db63-c492-433f-810c-2e58ea1368cf", "id": 3}] 2024-03-29T11:35:06.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.331+0000 7f0469f72640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "42e5db63-c492-433f-810c-2e58ea1368cf", "id": 3}] 2024-03-29T11:35:06.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.331+0000 7f0469f72640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.329 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.331+0000 7f0469f72640 1 -- 172.21.15.203:0/417689025 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "42e5db63-c492-433f-810c-2e58ea1368cf", "id": 3} v 0) v1 -- 0x7f04480058b0 con 0x7f046409ac40 2024-03-29T11:35:06.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.336+0000 7f0460ff9640 1 -- 172.21.15.203:0/417689025 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "42e5db63-c492-433f-810c-2e58ea1368cf", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 77944084 0 525046306) 0x7f0454006b00 con 0x7f046409ac40 2024-03-29T11:35:06.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.336+0000 7f0460ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "42e5db63-c492-433f-810c-2e58ea1368cf", "id": 3}] 2024-03-29T11:35:06.334 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.336+0000 7f0460ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:06.334 INFO:teuthology.orchestra.run.smithi203.stdout:3 2024-03-29T11:35:06.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 10 librados: watch_flush enter 2024-03-29T11:35:06.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 10 librados: watch_flush exit 2024-03-29T11:35:06.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-29T11:35:06.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 10 client.4129.objecter successfully canceled tick 2024-03-29T11:35:06.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 10 monclient: shutdown 2024-03-29T11:35:06.335 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:06.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 1 -- 172.21.15.203:0/417689025 >> v1:172.21.15.203:6789/0 conn(0x7f046409ac40 legacy=0x7f046413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 1 -- 172.21.15.203:0/417689025 shutdown_connections 2024-03-29T11:35:06.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 1 -- 172.21.15.203:0/417689025 >> 172.21.15.203:0/417689025 conn(0x7f0464066b00 msgr2=0x7f046413cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:06.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 1 -- 172.21.15.203:0/417689025 shutdown_connections 2024-03-29T11:35:06.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 1 -- 172.21.15.203:0/417689025 wait complete. 2024-03-29T11:35:06.336 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.337+0000 7f0469f72640 1 librados: shutdown 2024-03-29T11:35:06.345 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 3ff5f6e2-6f34-43d2-b346-4750c70d20f6 4 2024-03-29T11:35:06.386 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.388+0000 7fe599e1e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:35:06.397 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.400+0000 7fa732632040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:35:06.454 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.456+0000 7fe599e1e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:06.471 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.474+0000 7fa732632040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:06.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.489+0000 7efe22582640 10 monclient: get_monmap_and_config 2024-03-29T11:35:06.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.489+0000 7efe22582640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.487 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.489+0000 7efe22582640 10 monclient: monmap: 2024-03-29T11:35:06.487 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.487 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.487 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:35:06.490602+0000 2024-03-29T11:35:06.487 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:35:06.490602+0000 2024-03-29T11:35:06.488 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.488 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.488 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.488 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:06.488 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:06.488 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.490+0000 7efe22582640 1 Processor -- start 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.490+0000 7efe22582640 1 -- start start 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.490+0000 7efe22582640 10 monclient: init 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.490+0000 7efe22582640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 10 monclient(hunting): picked mon.noname-c con 0x7efe1c09ac40 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 10 monclient(hunting): picked mon.noname-a con 0x7efe1c09c870 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 10 monclient(hunting): picked mon.noname-b con 0x7efe1c0b4360 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c09dec0 con 0x7efe1c09c870 2024-03-29T11:35:06.497 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c09da30 con 0x7efe1c09ac40 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c0b39c0 con 0x7efe1c0b4360 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 10 monclient(hunting): authenticate will time out at 883.665935s 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7efe1c09ac40 0x7efe1c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45490/0 (socket says 172.21.15.203:45490) 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1bfff640 1 -- 172.21.15.203:0/938820305 learned_addr learned my addr 172.21.15.203:0/938820305 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666042523 0 0) 0x7efe1c09da30 con 0x7efe1c09ac40 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe04003520 con 0x7efe1c09ac40 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951385097 0 0) 0x7efe0c001110 con 0x7efe1c09c870 2024-03-29T11:35:06.498 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1c09da30 con 0x7efe1c09c870 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031945340 0 0) 0x7efe1c0b39c0 con 0x7efe1c0b4360 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe0c001110 con 0x7efe1c0b4360 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1695467001 0 0) 0x7efe04003520 con 0x7efe1c09ac40 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 >> v1:172.21.15.145:6790/0 conn(0x7efe1c0b4360 legacy=0x7efe1c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 >> v1:172.21.15.145:6789/0 conn(0x7efe1c09c870 legacy=0x7efe1c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.499 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 monclient: found mon.noname-c 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c0b5160 con 0x7efe1c09ac40 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe10001690 con 0x7efe1c09ac40 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: dump: 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.500 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe10001870 con 0x7efe1c09ac40 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 1 -- 172.21.15.203:0/938820305 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe10001c80 con 0x7efe1c09ac40 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe22582640 4 set_mon_vals no callback set 2024-03-29T11:35:06.501 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.491+0000 7efe1affd640 10 monclient: dump: 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.492+0000 7efe22582640 10 monclient: shutdown 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.492+0000 7efe22582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.492+0000 7efe22582640 1 -- 172.21.15.203:0/938820305 >> v1:172.21.15.203:6789/0 conn(0x7efe1c09ac40 legacy=0x7efe1c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.502 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.492+0000 7efe22582640 1 -- 172.21.15.203:0/938820305 shutdown_connections 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.492+0000 7efe22582640 1 -- 172.21.15.203:0/938820305 >> 172.21.15.203:0/938820305 conn(0x7efe1c066b00 msgr2=0x7efe1c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.492+0000 7efe22582640 1 -- 172.21.15.203:0/938820305 shutdown_connections 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.492+0000 7efe22582640 1 -- 172.21.15.203:0/938820305 wait complete. 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.493+0000 7efe22582640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.493+0000 7efe22582640 10 monclient: monmap: 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:06.503 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.493+0000 7efe22582640 1 librados: starting msgr at 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.493+0000 7efe22582640 1 librados: starting objecter 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.493+0000 7efe22582640 1 Processor -- start 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.493+0000 7efe22582640 1 -- start start 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.493+0000 7efe22582640 1 librados: setting wanted keys 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.493+0000 7efe22582640 1 librados: calling monclient init 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.493+0000 7efe22582640 10 monclient: init 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 10 monclient(hunting): picked mon.noname-b con 0x7efe1c09ac40 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 10 monclient(hunting): picked mon.noname-c con 0x7efe1c1ccb70 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 10 monclient(hunting): picked mon.noname-a con 0x7efe1c1425f0 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c1cd520 con 0x7efe1c1425f0 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c0acd00 con 0x7efe1c09ac40 2024-03-29T11:35:06.504 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe1c1cd000 con 0x7efe1c1ccb70 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 10 monclient(hunting): authenticate will time out at 883.669000s 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe1bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7efe1c09ac40 0x7efe1c1da690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45494/0 (socket says 172.21.15.203:45494) 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe1bfff640 1 -- 172.21.15.203:0/610888625 learned_addr learned my addr 172.21.15.203:0/610888625 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 client.?.objecter ms_handle_connect 0x7efe1c09ac40 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 client.?.objecter ms_handle_connect 0x7efe1c1ccb70 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 client.?.objecter ms_handle_connect 0x7efe1c1425f0 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424122953 0 0) 0x7efe1c0acd00 con 0x7efe1c09ac40 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.505 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe08004eb0 con 0x7efe1c09ac40 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857583287 0 0) 0x7efe1c1cd000 con 0x7efe1c1ccb70 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1c0acd00 con 0x7efe1c1ccb70 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768266920 0 0) 0x7efe1c1cd520 con 0x7efe1c1425f0 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe1c1cd000 con 0x7efe1c1425f0 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4108703843 0 0) 0x7efe08004eb0 con 0x7efe1c09ac40 2024-03-29T11:35:06.506 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe1c1cd520 con 0x7efe1c09ac40 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe10001bc0 con 0x7efe1c09ac40 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:06.507 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 950740930 0 0) 0x7efe1c1cd000 con 0x7efe1c1425f0 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe08004eb0 con 0x7efe1c1425f0 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe14001b50 con 0x7efe1c1425f0 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.508 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1551035648 0 0) 0x7efe1c0acd00 con 0x7efe1c1ccb70 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe1c1cd000 con 0x7efe1c1ccb70 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe0c0018a0 con 0x7efe1c1ccb70 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.509 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3411784224 0 0) 0x7efe08004eb0 con 0x7efe1c1425f0 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 >> v1:172.21.15.145:6790/0 conn(0x7efe1c1ccb70 legacy=0x7efe1c13adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 >> v1:172.21.15.203:6789/0 conn(0x7efe1c09ac40 legacy=0x7efe1c1da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 monclient: found mon.a 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1c0b1af0 con 0x7efe1c1425f0 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 5 monclient: authenticate success, global_id 4140 2024-03-29T11:35:06.510 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 10 monclient: _renew_subs 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.494+0000 7efe22582640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe1c1cc300 con 0x7efe1c1425f0 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe22582640 10 client.4140.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe22582640 10 monclient: _renew_subs 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe22582640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe22582640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe1c1daf80 con 0x7efe1c1425f0 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe22582640 1 librados: init done 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe14001f60 con 0x7efe1c1425f0 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe14003b30 con 0x7efe1c1425f0 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe1a7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 10 monclient: dump: 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.511 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7efe14004020 con 0x7efe1c1425f0 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 739145472 0 0) 0x7efe140047e0 con 0x7efe1c1425f0 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 10 client.4140.objecter ms_dispatch 0x7efe1c000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 3 client.4140.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 3 client.4140.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.495+0000 7efe18ff9640 20 client.4140.objecter dump_active .. 0 homeless 2024-03-29T11:35:06.512 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.496+0000 7efe22582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.496+0000 7efe22582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.496+0000 7efe22582640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:06.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.496+0000 7efe22582640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdfc005080 con 0x7efe1c1425f0 2024-03-29T11:35:06.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.498+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7efe14004b00 con 0x7efe1c1425f0 2024-03-29T11:35:06.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.498+0000 7efe18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.513 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.498+0000 7efe18ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:06.513 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.502+0000 7fe599e1e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:35:06.520 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.523+0000 7fa732632040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:35:06.553 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.555+0000 7fe599e1e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:35:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.571+0000 7efe22582640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3ff5f6e2-6f34-43d2-b346-4750c70d20f6", "id": 4}] 2024-03-29T11:35:06.568 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.571+0000 7efe22582640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3ff5f6e2-6f34-43d2-b346-4750c70d20f6", "id": 4}] 2024-03-29T11:35:06.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.571+0000 7efe22582640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:06.569 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.571+0000 7efe22582640 1 -- 172.21.15.203:0/610888625 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3ff5f6e2-6f34-43d2-b346-4750c70d20f6", "id": 4} v 0) v1 -- 0x7efdfc0058b0 con 0x7efe1c1425f0 2024-03-29T11:35:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.572+0000 7efe18ff9640 1 -- 172.21.15.203:0/610888625 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ff5f6e2-6f34-43d2-b346-4750c70d20f6", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 118332619 0 1697012583) 0x7efe140042d0 con 0x7efe1c1425f0 2024-03-29T11:35:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.572+0000 7efe18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3ff5f6e2-6f34-43d2-b346-4750c70d20f6", "id": 4}] 2024-03-29T11:35:06.570 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.572+0000 7efe18ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:06.570 INFO:teuthology.orchestra.run.smithi203.stdout:4 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 10 librados: watch_flush enter 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 10 librados: watch_flush exit 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 20 client.4140.objecter shutdown clearing up homeless session... 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 10 client.4140.objecter successfully canceled tick 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 10 monclient: shutdown 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 1 -- 172.21.15.203:0/610888625 >> v1:172.21.15.145:6789/0 conn(0x7efe1c1425f0 legacy=0x7efe1c1e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 1 -- 172.21.15.203:0/610888625 shutdown_connections 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 1 -- 172.21.15.203:0/610888625 >> 172.21.15.203:0/610888625 conn(0x7efe1c066b00 msgr2=0x7efe1c066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 1 -- 172.21.15.203:0/610888625 shutdown_connections 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 1 -- 172.21.15.203:0/610888625 wait complete. 2024-03-29T11:35:06.572 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.573+0000 7efe22582640 1 librados: shutdown 2024-03-29T11:35:06.574 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.577+0000 7fa732632040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:35:06.581 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 655ce144-bcb6-4dce-9e52-fd2a51349f52 5 2024-03-29T11:35:06.601 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.603+0000 7fe599e1e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:35:06.624 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:06.626+0000 7fa732632040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:35:06.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.724+0000 7ffbb1ec8640 10 monclient: get_monmap_and_config 2024-03-29T11:35:06.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.724+0000 7ffbb1ec8640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.722 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.724+0000 7ffbb1ec8640 10 monclient: monmap: 2024-03-29T11:35:06.722 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.722 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.722 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:35:06.725213+0000 2024-03-29T11:35:06.722 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:35:06.725213+0000 2024-03-29T11:35:06.722 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.722 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.723 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.723 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:06.723 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:06.723 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.724+0000 7ffbb1ec8640 1 Processor -- start 2024-03-29T11:35:06.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 1 -- start start 2024-03-29T11:35:06.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 10 monclient: init 2024-03-29T11:35:06.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:06.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 10 monclient(hunting): picked mon.noname-c con 0x7ffbac09ac40 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 10 monclient(hunting): picked mon.noname-b con 0x7ffbac09c870 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.732 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 10 monclient(hunting): picked mon.noname-a con 0x7ffbac0b4360 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac09dec0 con 0x7ffbac0b4360 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac09da30 con 0x7ffbac09ac40 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac0b39c0 con 0x7ffbac09c870 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbb1ec8640 10 monclient(hunting): authenticate will time out at 883.900569s 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbab7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ffbac09ac40 0x7ffbac0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45498/0 (socket says 172.21.15.203:45498) 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.725+0000 7ffbab7fe640 1 -- 172.21.15.203:0/3551501162 learned_addr learned my addr 172.21.15.203:0/3551501162 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1380872776 0 0) 0x7ffbac09da30 con 0x7ffbac09ac40 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb94003520 con 0x7ffbac09ac40 2024-03-29T11:35:06.733 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638066115 0 0) 0x7ffba40013c0 con 0x7ffbac0b4360 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbac09da30 con 0x7ffbac0b4360 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769635660 0 0) 0x7ffbac0b39c0 con 0x7ffbac09c870 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba40013c0 con 0x7ffbac09c870 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266829524 0 0) 0x7ffb94003520 con 0x7ffbac09ac40 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 >> v1:172.21.15.145:6790/0 conn(0x7ffbac09c870 legacy=0x7ffbac13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.734 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 >> v1:172.21.15.145:6789/0 conn(0x7ffbac0b4360 legacy=0x7ffbac14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 monclient: found mon.noname-c 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac0b5160 con 0x7ffbac09ac40 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ffb9c001690 con 0x7ffbac09ac40 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbb1ec8640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbb1ec8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: dump: 2024-03-29T11:35:06.735 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbb1ec8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb9c0017f0 con 0x7ffbac09ac40 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 1 -- 172.21.15.203:0/3551501162 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ffb9c001c40 con 0x7ffbac09ac40 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.736 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbaa7fc640 10 monclient: dump: 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbb1ec8640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.726+0000 7ffbb1ec8640 4 set_mon_vals no callback set 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.727+0000 7ffbb1ec8640 10 monclient: shutdown 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.727+0000 7ffbb1ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:06.737 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.727+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3551501162 >> v1:172.21.15.203:6789/0 conn(0x7ffbac09ac40 legacy=0x7ffbac0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.727+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3551501162 shutdown_connections 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.727+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3551501162 >> 172.21.15.203:0/3551501162 conn(0x7ffbac066b00 msgr2=0x7ffbac066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.727+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3551501162 shutdown_connections 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.727+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3551501162 wait complete. 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.728+0000 7ffbb1ec8640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.728+0000 7ffbb1ec8640 10 monclient: monmap: 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.738 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.728+0000 7ffbb1ec8640 1 librados: starting msgr at 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.728+0000 7ffbb1ec8640 1 librados: starting objecter 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.728+0000 7ffbb1ec8640 1 Processor -- start 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 1 -- start start 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 1 librados: setting wanted keys 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 1 librados: calling monclient init 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 10 monclient: init 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 10 monclient(hunting): picked mon.noname-c con 0x7ffbac09ac40 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 10 monclient(hunting): picked mon.noname-a con 0x7ffbac0b1110 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 10 monclient(hunting): picked mon.noname-b con 0x7ffbac1d7940 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.739 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac0b15a0 con 0x7ffbac0b1110 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac1d6d30 con 0x7ffbac1d7940 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.729+0000 7ffbb1ec8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbac13ffa0 con 0x7ffbac09ac40 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 10 monclient(hunting): authenticate will time out at 883.904834s 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbab7fe640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7ffbac09ac40 0x7ffbac0b03a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.203:56782/0 (socket says 172.21.15.203:56782) 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbabfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ffbac1d7940 0x7ffbac1e1850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45500/0 (socket says 172.21.15.203:45500) 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbab7fe640 1 -- 172.21.15.203:0/3124513693 learned_addr learned my addr 172.21.15.203:0/3124513693 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 client.?.objecter ms_handle_connect 0x7ffbac1d7940 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 client.?.objecter ms_handle_connect 0x7ffbac0b1110 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 client.?.objecter ms_handle_connect 0x7ffbac09ac40 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.740 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4003353713 0 0) 0x7ffbac1d6d30 con 0x7ffbac1d7940 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb98004eb0 con 0x7ffbac1d7940 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894370267 0 0) 0x7ffba40013c0 con 0x7ffbac0b1110 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbac1d6d30 con 0x7ffbac0b1110 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949102648 0 0) 0x7ffbac13ffa0 con 0x7ffbac09ac40 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.741 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba40013c0 con 0x7ffbac09ac40 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3250162516 0 0) 0x7ffb98004eb0 con 0x7ffbac1d7940 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbac13ffa0 con 0x7ffbac1d7940 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ffba4001d30 con 0x7ffbac1d7940 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): dump: 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.742 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1948782498 0 0) 0x7ffbac1d6d30 con 0x7ffbac0b1110 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb98004eb0 con 0x7ffbac0b1110 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ffba00016d0 con 0x7ffbac0b1110 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): dump: 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.743 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3360407512 0 0) 0x7ffba40013c0 con 0x7ffbac09ac40 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffbac1d6d30 con 0x7ffbac09ac40 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ffb9c001960 con 0x7ffbac09ac40 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient(hunting): dump: 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.744 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.745 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.745 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.745 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.745 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.745 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 132734415 0 0) 0x7ffbac13ffa0 con 0x7ffbac1d7940 2024-03-29T11:35:06.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 >> v1:172.21.15.145:6790/0 conn(0x7ffbac09ac40 legacy=0x7ffbac0b03a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.745 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 >> v1:172.21.15.145:6789/0 conn(0x7ffbac0b1110 legacy=0x7ffbac1d7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 monclient: found mon.b 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbac0ad870 con 0x7ffbac1d7940 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 10 monclient: _renew_subs 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac1e2f70 con 0x7ffbac1d7940 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:06.746 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 10 monclient: _renew_subs 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac1e32e0 con 0x7ffbac1d7940 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffbb1ec8640 1 librados: init done 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffba4003970 con 0x7ffbac1d7940 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ffba4003d80 con 0x7ffbac1d7940 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffba9ffb640 4 set_mon_vals no callback set 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 monclient: dump: 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.747 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ffba4004330 con 0x7ffbac1d7940 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 301571234 0 0) 0x7ffba4006bb0 con 0x7ffbac1d7940 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 10 client.4141.objecter ms_dispatch 0x7ffbac000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.730+0000 7ffb93fff640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.731+0000 7ffb93fff640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.731+0000 7ffbb1ec8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.731+0000 7ffbb1ec8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.731+0000 7ffbb1ec8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.748 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.731+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8c005080 con 0x7ffbac1d7940 2024-03-29T11:35:06.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.733+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ffba4006630 con 0x7ffbac1d7940 2024-03-29T11:35:06.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.733+0000 7ffb93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.749 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.733+0000 7ffb93fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:06.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.805+0000 7ffbb1ec8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "655ce144-bcb6-4dce-9e52-fd2a51349f52", "id": 5}] 2024-03-29T11:35:06.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.805+0000 7ffbb1ec8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "655ce144-bcb6-4dce-9e52-fd2a51349f52", "id": 5}] 2024-03-29T11:35:06.803 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.805+0000 7ffbb1ec8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.804 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.805+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3124513693 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "655ce144-bcb6-4dce-9e52-fd2a51349f52", "id": 5} v 0) v1 -- 0x7ffb8c0058b0 con 0x7ffbac1d7940 2024-03-29T11:35:06.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.809+0000 7ffb93fff640 1 -- 172.21.15.203:0/3124513693 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "655ce144-bcb6-4dce-9e52-fd2a51349f52", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3447536962 0 1988412176) 0x7ffba4006850 con 0x7ffbac1d7940 2024-03-29T11:35:06.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.809+0000 7ffb93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "655ce144-bcb6-4dce-9e52-fd2a51349f52", "id": 5}] 2024-03-29T11:35:06.807 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.809+0000 7ffb93fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:06.807 INFO:teuthology.orchestra.run.smithi203.stdout:5 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.810+0000 7ffbb1ec8640 10 librados: watch_flush enter 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.810+0000 7ffbb1ec8640 10 librados: watch_flush exit 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.810+0000 7ffbb1ec8640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.810+0000 7ffbb1ec8640 10 client.4141.objecter successfully canceled tick 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.810+0000 7ffbb1ec8640 10 monclient: shutdown 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.810+0000 7ffbb1ec8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.810+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3124513693 >> v1:172.21.15.203:6789/0 conn(0x7ffbac1d7940 legacy=0x7ffbac1e1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.810+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3124513693 shutdown_connections 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.810+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3124513693 >> 172.21.15.203:0/3124513693 conn(0x7ffbac066b00 msgr2=0x7ffbac0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.811+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3124513693 shutdown_connections 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.811+0000 7ffbb1ec8640 1 -- 172.21.15.203:0/3124513693 wait complete. 2024-03-29T11:35:06.809 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.811+0000 7ffbb1ec8640 1 librados: shutdown 2024-03-29T11:35:06.818 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 72515dad-d50c-4e13-bfc2-6ea629fbde5a 6 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.921+0000 7f9fda80f640 10 monclient: get_monmap_and_config 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.921+0000 7f9fda80f640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.921+0000 7f9fda80f640 10 monclient: monmap: 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:35:06.922047+0000 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:35:06.922047+0000 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:06.919 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:06.920 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.928 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.921+0000 7f9fda80f640 1 Processor -- start 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.921+0000 7f9fda80f640 1 -- start start 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.921+0000 7f9fda80f640 10 monclient: init 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 10 monclient(hunting): picked mon.noname-b con 0x7f9fd409ac40 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 10 monclient(hunting): picked mon.noname-c con 0x7f9fd409c870 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 10 monclient(hunting): picked mon.noname-a con 0x7f9fd40b4360 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd409dec0 con 0x7f9fd40b4360 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd409da30 con 0x7f9fd409c870 2024-03-29T11:35:06.929 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd40b39c0 con 0x7f9fd409ac40 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fda80f640 10 monclient(hunting): authenticate will time out at 884.097281s 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd37fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9fd409c870 0x7f9fd413f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45516/0 (socket says 172.21.15.203:45516) 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd37fe640 1 -- 172.21.15.203:0/1215865327 learned_addr learned my addr 172.21.15.203:0/1215865327 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2369347736 0 0) 0x7f9fd409da30 con 0x7f9fd409c870 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fbc003520 con 0x7f9fd409c870 2024-03-29T11:35:06.930 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436257763 0 0) 0x7f9fcc001490 con 0x7f9fd40b4360 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd409da30 con 0x7f9fd40b4360 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542809202 0 0) 0x7f9fd40b39c0 con 0x7f9fd409ac40 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.922+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fcc001490 con 0x7f9fd409ac40 2024-03-29T11:35:06.931 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 578682973 0 0) 0x7f9fbc003520 con 0x7f9fd409c870 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 >> v1:172.21.15.145:6790/0 conn(0x7f9fd409ac40 legacy=0x7f9fd40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 >> v1:172.21.15.145:6789/0 conn(0x7f9fd40b4360 legacy=0x7f9fd414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 1 monclient: found mon.noname-c 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd40b5160 con 0x7f9fd409c870 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:35:06.932 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9fc4001690 con 0x7f9fd409c870 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: dump: 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.933 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fc4001870 con 0x7f9fd409c870 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 1 -- 172.21.15.203:0/1215865327 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9fc4001c80 con 0x7f9fd409c870 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 4 set_mon_vals no callback set 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fd2ffd640 10 monclient: dump: 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.934 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 10 monclient: shutdown 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 1 -- 172.21.15.203:0/1215865327 >> v1:172.21.15.203:6789/0 conn(0x7f9fd409c870 legacy=0x7f9fd413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 1 -- 172.21.15.203:0/1215865327 shutdown_connections 2024-03-29T11:35:06.935 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 1 -- 172.21.15.203:0/1215865327 >> 172.21.15.203:0/1215865327 conn(0x7f9fd4066b00 msgr2=0x7f9fd4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 1 -- 172.21.15.203:0/1215865327 shutdown_connections 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.923+0000 7f9fda80f640 1 -- 172.21.15.203:0/1215865327 wait complete. 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.924+0000 7f9fda80f640 10 monclient: build_initial_monmap 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.924+0000 7f9fda80f640 10 monclient: monmap: 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:06.936 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.924+0000 7f9fda80f640 1 librados: starting msgr at 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.924+0000 7f9fda80f640 1 librados: starting objecter 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.924+0000 7f9fda80f640 1 Processor -- start 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 1 -- start start 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 1 librados: setting wanted keys 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 1 librados: calling monclient init 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 10 monclient: init 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 10 monclient(hunting): picked mon.noname-c con 0x7f9fd409ac40 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 10 monclient(hunting): picked mon.noname-b con 0x7f9fd40b6b00 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:06.937 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 10 monclient(hunting): picked mon.noname-a con 0x7f9fd41406b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd41d7e20 con 0x7f9fd41406b0 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd4140b00 con 0x7f9fd40b6b00 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fd40ad080 con 0x7f9fd409ac40 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fda80f640 10 monclient(hunting): authenticate will time out at 884.100451s 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd37fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9fd40b6b00 0x7f9fd413ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45524/0 (socket says 172.21.15.203:45524) 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd37fe640 1 -- 172.21.15.203:0/2154702177 learned_addr learned my addr 172.21.15.203:0/2154702177 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f9fd40b6b00 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f9fd409ac40 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.938 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 client.?.objecter ms_handle_connect 0x7f9fd41406b0 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509662384 0 0) 0x7f9fd4140b00 con 0x7f9fd40b6b00 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fc0004eb0 con 0x7f9fd40b6b00 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504895457 0 0) 0x7f9fd40ad080 con 0x7f9fd409ac40 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.939 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd4140b00 con 0x7f9fd409ac40 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.925+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456993212 0 0) 0x7f9fd41d7e20 con 0x7f9fd41406b0 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd40ad080 con 0x7f9fd41406b0 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357252694 0 0) 0x7f9fc0004eb0 con 0x7f9fd40b6b00 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fd41d7e20 con 0x7f9fd40b6b00 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9fc4001bc0 con 0x7f9fd40b6b00 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.940 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.941 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 836303568 0 0) 0x7f9fd4140b00 con 0x7f9fd409ac40 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fc0004eb0 con 0x7f9fd409ac40 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9fc8001870 con 0x7f9fd409ac40 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.942 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1349635378 0 0) 0x7f9fd40ad080 con 0x7f9fd41406b0 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fd4140b00 con 0x7f9fd41406b0 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9fcc001a00 con 0x7f9fd41406b0 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.943 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3795780264 0 0) 0x7f9fd41d7e20 con 0x7f9fd40b6b00 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 >> v1:172.21.15.145:6790/0 conn(0x7f9fd409ac40 legacy=0x7f9fd41d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 >> v1:172.21.15.145:6789/0 conn(0x7f9fd41406b0 legacy=0x7f9fd41e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:06.944 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 monclient: found mon.b 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd41d7ab0 con 0x7f9fd40b6b00 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fda80f640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fda80f640 10 monclient: _renew_subs 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fda80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fda80f640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fd41e3570 con 0x7f9fd40b6b00 2024-03-29T11:35:06.945 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fda80f640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fda80f640 10 monclient: _renew_subs 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fda80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fda80f640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fd41e38e0 con 0x7f9fd40b6b00 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fda80f640 1 librados: init done 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fc4004bf0 con 0x7f9fd40b6b00 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9fc4004f00 con 0x7f9fd40b6b00 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd27fc640 4 set_mon_vals no callback set 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 monclient: dump: 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:06.946 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9fc40053f0 con 0x7f9fd40b6b00 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2861769295 0 0) 0x7f9fc4005c90 con 0x7f9fd40b6b00 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 10 client.4147.objecter ms_dispatch 0x7f9fd4000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:35:06.947 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:35:06.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.926+0000 7f9fd0ff9640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-29T11:35:06.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.927+0000 7f9fda80f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.927+0000 7f9fda80f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.927+0000 7f9fda80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:06.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.927+0000 7f9fda80f640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb4005080 con 0x7f9fd40b6b00 2024-03-29T11:35:06.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.929+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9fc4005f70 con 0x7f9fd40b6b00 2024-03-29T11:35:06.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.929+0000 7f9fd0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:06.948 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:06.929+0000 7f9fd0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:06.984 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:06.986+0000 7fe599e1e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:35:07.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.002+0000 7f9fda80f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "72515dad-d50c-4e13-bfc2-6ea629fbde5a", "id": 6}] 2024-03-29T11:35:07.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.002+0000 7f9fda80f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "72515dad-d50c-4e13-bfc2-6ea629fbde5a", "id": 6}] 2024-03-29T11:35:07.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.002+0000 7f9fda80f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:07.000 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.002+0000 7f9fda80f640 1 -- 172.21.15.203:0/2154702177 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "72515dad-d50c-4e13-bfc2-6ea629fbde5a", "id": 6} v 0) v1 -- 0x7f9fb40058b0 con 0x7f9fd40b6b00 2024-03-29T11:35:07.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.006+0000 7f9fd0ff9640 1 -- 172.21.15.203:0/2154702177 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "72515dad-d50c-4e13-bfc2-6ea629fbde5a", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2463805203 0 1113811849) 0x7f9fc4005770 con 0x7f9fd40b6b00 2024-03-29T11:35:07.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.006+0000 7f9fd0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "72515dad-d50c-4e13-bfc2-6ea629fbde5a", "id": 6}] 2024-03-29T11:35:07.004 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.006+0000 7f9fd0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:07.004 INFO:teuthology.orchestra.run.smithi203.stdout:6 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 10 librados: watch_flush enter 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 10 librados: watch_flush exit 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 10 client.4147.objecter successfully canceled tick 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 10 monclient: shutdown 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 1 -- 172.21.15.203:0/2154702177 >> v1:172.21.15.203:6789/0 conn(0x7f9fd40b6b00 legacy=0x7f9fd413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 1 -- 172.21.15.203:0/2154702177 shutdown_connections 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 1 -- 172.21.15.203:0/2154702177 >> 172.21.15.203:0/2154702177 conn(0x7f9fd4066b00 msgr2=0x7f9fd40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 1 -- 172.21.15.203:0/2154702177 shutdown_connections 2024-03-29T11:35:07.006 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 1 -- 172.21.15.203:0/2154702177 wait complete. 2024-03-29T11:35:07.007 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.007+0000 7f9fda80f640 1 librados: shutdown 2024-03-29T11:35:07.015 DEBUG:teuthology.orchestra.run.smithi203:> sudo ceph --cluster ceph osd new 8a0b7d49-f607-418c-9c4e-44e6f25cd916 7 2024-03-29T11:35:07.032 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:07.035+0000 7fa732632040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.117+0000 7fd17c64a640 10 monclient: get_monmap_and_config 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.117+0000 7fd17c64a640 10 monclient: build_initial_monmap 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.117+0000 7fd17c64a640 10 monclient: monmap: 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:35:07.118741+0000 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:35:07.118741+0000 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:07.116 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:07.117 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:07.117 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:07.117 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:07.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.118+0000 7fd17c64a640 1 Processor -- start 2024-03-29T11:35:07.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.118+0000 7fd17c64a640 1 -- start start 2024-03-29T11:35:07.117 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.118+0000 7fd17c64a640 10 monclient: init 2024-03-29T11:35:07.123 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:07.121+0000 7fe599e1e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:35:07.127 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 10 monclient(hunting): picked mon.noname-a con 0x7fd17409ac40 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 10 monclient(hunting): picked mon.noname-b con 0x7fd17409c870 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 10 monclient(hunting): picked mon.noname-c con 0x7fd1740b4360 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17409dec0 con 0x7fd17409ac40 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd17409da30 con 0x7fd1740b4360 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740b39c0 con 0x7fd17409c870 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 10 monclient(hunting): authenticate will time out at 884.294153s 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17abe7640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd1740b4360 0x7fd17414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45538/0 (socket says 172.21.15.203:45538) 2024-03-29T11:35:07.128 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17abe7640 1 -- 172.21.15.203:0/3308145221 learned_addr learned my addr 172.21.15.203:0/3308145221 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501469568 0 0) 0x7fd17409da30 con 0x7fd1740b4360 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd160003520 con 0x7fd1740b4360 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307354192 0 0) 0x7fd164001110 con 0x7fd17409ac40 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd17409da30 con 0x7fd17409ac40 2024-03-29T11:35:07.129 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3310888747 0 0) 0x7fd1740b39c0 con 0x7fd17409c870 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd164001110 con 0x7fd17409c870 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 253891992 0 0) 0x7fd160003520 con 0x7fd1740b4360 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 >> v1:172.21.15.145:6790/0 conn(0x7fd17409c870 legacy=0x7fd17413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 >> v1:172.21.15.145:6789/0 conn(0x7fd17409ac40 legacy=0x7fd1740b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient: _finish_hunting 0 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 monclient: found mon.noname-c 2024-03-29T11:35:07.130 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740b5160 con 0x7fd1740b4360 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient: _finish_auth 0 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient: _check_auth_tickets 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd16c001a10 con 0x7fd1740b4360 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd1793e4640 10 monclient: dump: 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:07.131 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.119+0000 7fd17c64a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd16c001b70 con 0x7fd1740b4360 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd1793e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:07.132 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd17c64a640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd1793e4640 1 -- 172.21.15.203:0/3308145221 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd16c001fc0 con 0x7fd1740b4360 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd1793e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd17c64a640 4 set_mon_vals no callback set 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd1793e4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd1793e4640 10 monclient: dump: 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.133 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd17c64a640 10 monclient: shutdown 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd17c64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd17c64a640 1 -- 172.21.15.203:0/3308145221 >> v1:172.21.15.203:6789/0 conn(0x7fd1740b4360 legacy=0x7fd17414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd17c64a640 1 -- 172.21.15.203:0/3308145221 shutdown_connections 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd17c64a640 1 -- 172.21.15.203:0/3308145221 >> 172.21.15.203:0/3308145221 conn(0x7fd174066b00 msgr2=0x7fd174066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd17c64a640 1 -- 172.21.15.203:0/3308145221 shutdown_connections 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.120+0000 7fd17c64a640 1 -- 172.21.15.203:0/3308145221 wait complete. 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.121+0000 7fd17c64a640 10 monclient: build_initial_monmap 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.121+0000 7fd17c64a640 10 monclient: monmap: 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 0 2024-03-29T11:35:07.134 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 0.000000 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:created 0.000000 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.121+0000 7fd17c64a640 1 librados: starting msgr at 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.121+0000 7fd17c64a640 1 librados: starting objecter 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.121+0000 7fd17c64a640 1 Processor -- start 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.121+0000 7fd17c64a640 1 -- start start 2024-03-29T11:35:07.135 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.121+0000 7fd17c64a640 1 librados: setting wanted keys 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.121+0000 7fd17c64a640 1 librados: calling monclient init 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.121+0000 7fd17c64a640 10 monclient: init 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 10 monclient(hunting): picked mon.noname-c con 0x7fd17409ac40 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 10 monclient(hunting): picked mon.noname-b con 0x7fd1740b1070 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 10 monclient(hunting): picked mon.noname-a con 0x7fd1740acf70 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd174144c00 con 0x7fd1740acf70 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1740ae870 con 0x7fd1740b1070 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1741397e0 con 0x7fd17409ac40 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd17c64a640 10 monclient(hunting): authenticate will time out at 884.297120s 2024-03-29T11:35:07.136 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd179be5640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd1740b1070 0x7fd1741389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.203:45544/0 (socket says 172.21.15.203:45544) 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd179be5640 1 -- 172.21.15.203:0/1056517390 learned_addr learned my addr 172.21.15.203:0/1056517390 (peer_addr_for_me v1:172.21.15.203:0/0) 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 client.?.objecter ms_handle_connect 0x7fd1740b1070 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 client.?.objecter ms_handle_connect 0x7fd17409ac40 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 client.?.objecter ms_handle_connect 0x7fd1740acf70 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031039980 0 0) 0x7fd1740ae870 con 0x7fd1740b1070 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:35:07.137 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd158004eb0 con 0x7fd1740b1070 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135730210 0 0) 0x7fd1741397e0 con 0x7fd17409ac40 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): my global_id is 4160 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1740ae870 con 0x7fd17409ac40 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680115717 0 0) 0x7fd174144c00 con 0x7fd1740acf70 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:35:07.138 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1741397e0 con 0x7fd1740acf70 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 452607186 0 0) 0x7fd158004eb0 con 0x7fd1740b1070 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd174144c00 con 0x7fd1740b1070 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2637649606 0 0) 0x7fd1740ae870 con 0x7fd17409ac40 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd158004eb0 con 0x7fd17409ac40 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd170001a10 con 0x7fd1740b1070 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): dump: 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.139 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3435164459 0 0) 0x7fd1741397e0 con 0x7fd1740acf70 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.122+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1740ae870 con 0x7fd1740acf70 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd164001150 con 0x7fd17409ac40 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient(hunting): dump: 2024-03-29T11:35:07.140 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd16c002100 con 0x7fd1740acf70 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:07.141 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient(hunting): dump: 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2967873947 0 0) 0x7fd174144c00 con 0x7fd1740b1070 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 >> v1:172.21.15.145:6790/0 conn(0x7fd17409ac40 legacy=0x7fd1741d7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.142 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 >> v1:172.21.15.145:6789/0 conn(0x7fd1740acf70 legacy=0x7fd1741390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 monclient: found mon.b 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1740aeae0 con 0x7fd1740b1070 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd17c64a640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd17c64a640 10 monclient: _renew_subs 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd17c64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:07.143 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd17c64a640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1740b1b10 con 0x7fd1740b1070 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd17c64a640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd17c64a640 10 monclient: _renew_subs 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd17c64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd17c64a640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd174145ea0 con 0x7fd1740b1070 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd17c64a640 1 librados: init done 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd170001b70 con 0x7fd1740b1070 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd170003a40 con 0x7fd1740b1070 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 monclient: dump: 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:epoch 1 2024-03-29T11:35:07.144 INFO:teuthology.orchestra.run.smithi203.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:election_strategy: 1 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd178be3640 4 set_mon_vals no callback set 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd170003fb0 con 0x7fd1740b1070 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1560640658 0 0) 0x7fd170004260 con 0x7fd1740b1070 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 10 client.4153.objecter ms_dispatch 0x7fd174000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:35:07.145 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:35:07.146 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:35:07.146 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.123+0000 7fd16affd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-29T11:35:07.146 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.124+0000 7fd17c64a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:07.146 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.124+0000 7fd17c64a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:07.146 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.124+0000 7fd17c64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:07.146 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.124+0000 7fd17c64a640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd17409b020 con 0x7fd1740b1070 2024-03-29T11:35:07.146 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.127+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd170005780 con 0x7fd1740b1070 2024-03-29T11:35:07.146 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.127+0000 7fd16affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:07.146 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.127+0000 7fd16affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:07.176 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:07.178+0000 7fa732632040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:35:07.197 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.200+0000 7fd17c64a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8a0b7d49-f607-418c-9c4e-44e6f25cd916", "id": 7}] 2024-03-29T11:35:07.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.200+0000 7fd17c64a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8a0b7d49-f607-418c-9c4e-44e6f25cd916", "id": 7}] 2024-03-29T11:35:07.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.200+0000 7fd17c64a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:07.198 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.200+0000 7fd17c64a640 1 -- 172.21.15.203:0/1056517390 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8a0b7d49-f607-418c-9c4e-44e6f25cd916", "id": 7} v 0) v1 -- 0x7fd17404f730 con 0x7fd1740b1070 2024-03-29T11:35:07.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.204+0000 7fd16affd640 1 -- 172.21.15.203:0/1056517390 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8a0b7d49-f607-418c-9c4e-44e6f25cd916", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 787912125 0 1371665406) 0x7fd170005a20 con 0x7fd1740b1070 2024-03-29T11:35:07.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.204+0000 7fd16affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8a0b7d49-f607-418c-9c4e-44e6f25cd916", "id": 7}] 2024-03-29T11:35:07.202 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.204+0000 7fd16affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:07.202 INFO:teuthology.orchestra.run.smithi203.stdout:7 2024-03-29T11:35:07.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 10 librados: watch_flush enter 2024-03-29T11:35:07.203 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 10 librados: watch_flush exit 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 10 client.4153.objecter successfully canceled tick 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 10 monclient: shutdown 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 1 -- 172.21.15.203:0/1056517390 >> v1:172.21.15.203:6789/0 conn(0x7fd1740b1070 legacy=0x7fd1741389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 1 -- 172.21.15.203:0/1056517390 shutdown_connections 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 1 -- 172.21.15.203:0/1056517390 >> 172.21.15.203:0/1056517390 conn(0x7fd174066b00 msgr2=0x7fd17413cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 1 -- 172.21.15.203:0/1056517390 shutdown_connections 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 1 -- 172.21.15.203:0/1056517390 wait complete. 2024-03-29T11:35:07.204 INFO:teuthology.orchestra.run.smithi203.stderr:2024-03-29T11:35:07.205+0000 7fd17c64a640 1 librados: shutdown 2024-03-29T11:35:07.213 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:35:07.213 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:35:07.215 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:35:07.215 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:35:07.215 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:35:07.217 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:35:07.217 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:35:07.217 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:35:07.219 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:35:07.219 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:35:07.220 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:35:07.222 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:35:07.222 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:35:07.222 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:35:07.231 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:07.233+0000 7fe599e1e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:35:07.255 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:35:07.255 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:35:07.255 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:35:07.257 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:35:07.257 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:35:07.257 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:35:07.259 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:35:07.259 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:35:07.260 DEBUG:teuthology.orchestra.run.smithi203:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:35:07.261 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:35:07.262 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:07.306 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:07.308+0000 7fe599e1e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:35:07.325 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:07.328+0000 7fa732632040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:35:07.378 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.380+0000 7f436d515640 10 monclient: get_monmap_and_config 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.380+0000 7f436d515640 10 monclient: build_initial_monmap 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.380+0000 7f436d515640 10 monclient: monmap: 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 0 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:35:07.381973+0000 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:35:07.381973+0000 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:07.379 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:07.380 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:07.381+0000 7fa732632040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:35:07.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.381+0000 7f436d515640 1 Processor -- start 2024-03-29T11:35:07.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.381+0000 7f436d515640 1 -- start start 2024-03-29T11:35:07.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.381+0000 7f436d515640 10 monclient: init 2024-03-29T11:35:07.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.381+0000 7f436d515640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:07.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:07.388 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 10 monclient(hunting): picked mon.noname-a con 0x7f436809ac40 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 10 monclient(hunting): picked mon.noname-c con 0x7f436809c870 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 10 monclient(hunting): picked mon.noname-b con 0x7f43680b4360 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436809dec0 con 0x7f436809ac40 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436809da30 con 0x7f436809c870 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43680b39c0 con 0x7f43680b4360 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 10 monclient(hunting): authenticate will time out at 896.690108s 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43677fe640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f43680b4360 0x7f436814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:38504/0 (socket says 172.21.15.145:38504) 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f4366ffd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f436809ac40 0x7f43680b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36480/0 (socket says 172.21.15.145:36480) 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43677fe640 1 -- 172.21.15.145:0/4127965670 learned_addr learned my addr 172.21.15.145:0/4127965670 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 449296180 0 0) 0x7f4358001110 con 0x7f436809ac40 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.389 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4354003520 con 0x7f436809ac40 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239634579 0 0) 0x7f43680b39c0 con 0x7f43680b4360 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4358001110 con 0x7f43680b4360 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440257588 0 0) 0x7f436809da30 con 0x7f436809c870 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43680b39c0 con 0x7f436809c870 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 389179585 0 0) 0x7f4358001110 con 0x7f43680b4360 2024-03-29T11:35:07.390 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 >> v1:172.21.15.203:6789/0 conn(0x7f436809c870 legacy=0x7f436813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 >> v1:172.21.15.145:6789/0 conn(0x7f436809ac40 legacy=0x7f43680b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 monclient: found mon.noname-b 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43680b5160 con 0x7f43680b4360 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4360001920 con 0x7f43680b4360 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: dump: 2024-03-29T11:35:07.391 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4360001a80 con 0x7f43680b4360 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 1 -- 172.21.15.145:0/4127965670 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4360001ed0 con 0x7f43680b4360 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:07.392 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f436d515640 4 set_mon_vals no callback set 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.382+0000 7f43667fc640 10 monclient: dump: 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.383+0000 7f436d515640 10 monclient: shutdown 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.383+0000 7f436d515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:07.393 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.383+0000 7f436d515640 1 -- 172.21.15.145:0/4127965670 >> v1:172.21.15.145:6790/0 conn(0x7f43680b4360 legacy=0x7f436814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.383+0000 7f436d515640 1 -- 172.21.15.145:0/4127965670 shutdown_connections 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.383+0000 7f436d515640 1 -- 172.21.15.145:0/4127965670 >> 172.21.15.145:0/4127965670 conn(0x7f4368066b00 msgr2=0x7f4368066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.383+0000 7f436d515640 1 -- 172.21.15.145:0/4127965670 shutdown_connections 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.383+0000 7f436d515640 1 -- 172.21.15.145:0/4127965670 wait complete. 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.384+0000 7f436d515640 10 monclient: build_initial_monmap 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.384+0000 7f436d515640 10 monclient: monmap: 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 0 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:created 0.000000 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:07.394 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.384+0000 7f436d515640 1 librados: starting msgr at 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.384+0000 7f436d515640 1 librados: starting objecter 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.384+0000 7f436d515640 1 Processor -- start 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.384+0000 7f436d515640 1 -- start start 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.384+0000 7f436d515640 1 librados: setting wanted keys 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.384+0000 7f436d515640 1 librados: calling monclient init 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.384+0000 7f436d515640 10 monclient: init 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 10 monclient(hunting): picked mon.noname-b con 0x7f436809ac40 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 10 monclient(hunting): picked mon.noname-a con 0x7f43680b08d0 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 10 monclient(hunting): picked mon.noname-c con 0x7f43681d7c00 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4368140da0 con 0x7f43680b08d0 2024-03-29T11:35:07.395 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4368141010 con 0x7f436809ac40 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f43677fe640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f43681d7c00 0x7f4368136d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:38506/0 (socket says 172.21.15.145:38506) 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f43677fe640 1 -- 172.21.15.145:0/1159367305 learned_addr learned my addr 172.21.15.145:0/1159367305 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43681d6fd0 con 0x7f43681d7c00 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f436d515640 10 monclient(hunting): authenticate will time out at 896.693540s 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 client.?.objecter ms_handle_connect 0x7f43681d7c00 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 client.?.objecter ms_handle_connect 0x7f436809ac40 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 client.?.objecter ms_handle_connect 0x7f43680b08d0 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879291211 0 0) 0x7f43681d6fd0 con 0x7f43681d7c00 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.396 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4348004eb0 con 0x7f43681d7c00 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235310313 0 0) 0x7f4368141010 con 0x7f436809ac40 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43681d6fd0 con 0x7f436809ac40 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 496269072 0 0) 0x7f4368140da0 con 0x7f43680b08d0 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4368141010 con 0x7f43680b08d0 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4085589670 0 0) 0x7f4348004eb0 con 0x7f43681d7c00 2024-03-29T11:35:07.397 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4368140da0 con 0x7f43681d7c00 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4360001740 con 0x7f43681d7c00 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.385+0000 7f435f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient(hunting): dump: 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:07.398 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2916962961 0 0) 0x7f43681d6fd0 con 0x7f436809ac40 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4348004eb0 con 0x7f436809ac40 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 114138739 0 0) 0x7f4368141010 con 0x7f43680b08d0 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43681d6fd0 con 0x7f43680b08d0 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f43580012f0 con 0x7f436809ac40 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient(hunting): dump: 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:07.399 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4350001830 con 0x7f43680b08d0 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient(hunting): dump: 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:07.400 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 251949455 0 0) 0x7f4368140da0 con 0x7f43681d7c00 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 >> v1:172.21.15.203:6789/0 conn(0x7f436809ac40 legacy=0x7f43680b1350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 >> v1:172.21.15.145:6789/0 conn(0x7f43680b08d0 legacy=0x7f43681d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 monclient: found mon.c 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43680add50 con 0x7f43681d7c00 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f436d515640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:35:07.401 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f436d515640 10 monclient: _renew_subs 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f436d515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4368137470 con 0x7f43681d7c00 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4360004e10 con 0x7f43681d7c00 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f436d515640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f436d515640 10 monclient: _renew_subs 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f4365ffb640 4 set_mon_vals no callback set 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f436d515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43681e3900 con 0x7f43681d7c00 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f436d515640 1 librados: init done 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4360005120 con 0x7f43681d7c00 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 monclient: dump: 2024-03-29T11:35:07.402 INFO:teuthology.orchestra.run.smithi145.stderr:epoch 1 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4360005610 con 0x7f43681d7c00 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f4360006d70 con 0x7f43681d7c00 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 10 client.4190.objecter ms_dispatch 0x7f4368000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.386+0000 7f435f7fe640 20 client.4190.objecter dump_active .. 0 homeless 2024-03-29T11:35:07.403 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.387+0000 7f436d515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:07.404 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.387+0000 7f436d515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:07.404 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.387+0000 7f436d515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:07.404 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.387+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43681d7160 con 0x7f43681d7c00 2024-03-29T11:35:07.404 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.389+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4360006690 con 0x7f43681d7c00 2024-03-29T11:35:07.404 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.389+0000 7f435f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:07.404 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.389+0000 7f435f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:07.426 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:07.427+0000 7fe599e1e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:07.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.450+0000 7f436d515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:07.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.450+0000 7f436d515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:07.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.450+0000 7f436d515640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:07.448 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.450+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f436809b020 con 0x7f43681d7c00 2024-03-29T11:35:07.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.450+0000 7f435f7fe640 1 -- 172.21.15.145:0/1159367305 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f4360006940 con 0x7f43681d7c00 2024-03-29T11:35:07.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.450+0000 7f435f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:07.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.450+0000 7f435f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:07.449 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-03-29T11:35:07.449 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:07.450 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 10 librados: watch_flush enter 2024-03-29T11:35:07.450 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 10 librados: watch_flush exit 2024-03-29T11:35:07.450 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 20 client.4190.objecter shutdown clearing up homeless session... 2024-03-29T11:35:07.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 10 client.4190.objecter successfully canceled tick 2024-03-29T11:35:07.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 10 monclient: shutdown 2024-03-29T11:35:07.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:07.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 >> v1:172.21.15.145:6790/0 conn(0x7f43681d7c00 legacy=0x7f4368136d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:07.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 shutdown_connections 2024-03-29T11:35:07.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 >> 172.21.15.145:0/1159367305 conn(0x7f4368066b00 msgr2=0x7f436813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:07.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 shutdown_connections 2024-03-29T11:35:07.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 1 -- 172.21.15.145:0/1159367305 wait complete. 2024-03-29T11:35:07.451 INFO:teuthology.orchestra.run.smithi145.stderr:2024-03-29T11:35:07.451+0000 7f436d515640 1 librados: shutdown 2024-03-29T11:35:07.460 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:35:07.460 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:35:07.491 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:07.493+0000 7fa732632040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:07.524 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:07.526+0000 7fe599e1e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:35:07.586 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:07.589+0000 7fa732632040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:35:07.680 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:07.682+0000 7fe599e1e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:07.722 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:07.724+0000 7fe599e1e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:07.743 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:07.745+0000 7fa732632040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:35:07.786 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:07.788+0000 7fa732632040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:35:07.828 INFO:tasks.ceph.osd.1.smithi145.stderr:2024-03-29T11:35:07.830+0000 7f5429214640 -1 Falling back to public interface 2024-03-29T11:35:07.835 INFO:tasks.ceph.osd.2.smithi145.stderr:2024-03-29T11:35:07.837+0000 7fb528015640 -1 Falling back to public interface 2024-03-29T11:35:07.839 INFO:tasks.ceph.osd.3.smithi145.stderr:2024-03-29T11:35:07.841+0000 7f89c1811640 -1 Falling back to public interface 2024-03-29T11:35:07.843 INFO:tasks.ceph.osd.0.smithi145.stderr:2024-03-29T11:35:07.845+0000 7fb7728bd640 -1 Falling back to public interface 2024-03-29T11:35:07.859 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-29T11:35:07.861+0000 7f005dc1e640 -1 Falling back to public interface 2024-03-29T11:35:07.860 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-29T11:35:07.862+0000 7f719e1f8640 -1 Falling back to public interface 2024-03-29T11:35:07.866 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-29T11:35:07.868+0000 7f94aba17640 -1 Falling back to public interface 2024-03-29T11:35:07.866 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-29T11:35:07.868+0000 7efee03bd640 -1 Falling back to public interface 2024-03-29T11:35:07.878 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:07.880+0000 7fe599e1e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:35:07.945 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:07.948+0000 7fa732632040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:35:08.266 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:08.355 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.357+0000 7f8131f0e640 10 monclient: get_monmap_and_config 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.357+0000 7f8131f0e640 10 monclient: build_initial_monmap 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.357+0000 7f8131f0e640 10 monclient: monmap: 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:35:08.359025+0000 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:35:08.359025+0000 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:08.356 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.358+0000 7f8131f0e640 1 Processor -- start 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.358+0000 7f8131f0e640 1 -- start start 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.358+0000 7f8131f0e640 10 monclient: init 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f812c1457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f812c13f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f812c0a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c0aaef0 con 0x7f812c13f650 2024-03-29T11:35:08.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c145bd0 con 0x7f812c1457f0 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c145eb0 con 0x7f812c0a9d70 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 10 monclient(hunting): authenticate will time out at 897.667150s 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8122ffd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f812c13f650 0x7f812c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36508/0 (socket says 172.21.15.145:36508) 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8122ffd640 1 -- 172.21.15.145:0/840052746 learned_addr learned my addr 172.21.15.145:0/840052746 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812bfff640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f812c0a9d70 0x7f812c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:38522/0 (socket says 172.21.15.145:38522) 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551390540 0 0) 0x7f812c0aaef0 con 0x7f812c13f650 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f810c003520 con 0x7f812c13f650 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 919621104 0 0) 0x7f812c145eb0 con 0x7f812c0a9d70 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:08.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812c0aaef0 con 0x7f812c0a9d70 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375002475 0 0) 0x7f812c145bd0 con 0x7f812c1457f0 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812c145eb0 con 0x7f812c1457f0 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2255765689 0 0) 0x7f812c0aaef0 con 0x7f812c0a9d70 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 >> v1:172.21.15.203:6789/0 conn(0x7f812c1457f0 legacy=0x7f812c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 >> v1:172.21.15.145:6789/0 conn(0x7f812c13f650 legacy=0x7f812c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 monclient: found mon.noname-b 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:08.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c146340 con 0x7f812c0a9d70 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: _finish_auth 0 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8124001970 con 0x7f812c0a9d70 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 5 monclient: authenticate success, global_id 4193 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: dump: 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:08.368 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8124001ad0 con 0x7f812c0a9d70 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 1 -- 172.21.15.145:0/840052746 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8124001f20 con 0x7f812c0a9d70 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f8131f0e640 4 set_mon_vals no callback set 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.359+0000 7f812affd640 10 monclient: dump: 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:08.369 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.360+0000 7f8131f0e640 10 monclient: shutdown 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.360+0000 7f8131f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.360+0000 7f8131f0e640 1 -- 172.21.15.145:0/840052746 >> v1:172.21.15.145:6790/0 conn(0x7f812c0a9d70 legacy=0x7f812c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.360+0000 7f8131f0e640 1 -- 172.21.15.145:0/840052746 shutdown_connections 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.360+0000 7f8131f0e640 1 -- 172.21.15.145:0/840052746 >> 172.21.15.145:0/840052746 conn(0x7f812c066a00 msgr2=0x7f812c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:08.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.360+0000 7f8131f0e640 1 -- 172.21.15.145:0/840052746 shutdown_connections 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.360+0000 7f8131f0e640 1 -- 172.21.15.145:0/840052746 wait complete. 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.361+0000 7f8131f0e640 10 monclient: build_initial_monmap 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.361+0000 7f8131f0e640 10 monclient: monmap: 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.361+0000 7f8131f0e640 1 librados: starting msgr at 2024-03-29T11:35:08.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.361+0000 7f8131f0e640 1 librados: starting objecter 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.361+0000 7f8131f0e640 1 Processor -- start 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.361+0000 7f8131f0e640 1 -- start start 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.361+0000 7f8131f0e640 1 librados: setting wanted keys 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.361+0000 7f8131f0e640 1 librados: calling monclient init 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.361+0000 7f8131f0e640 10 monclient: init 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient(hunting): picked mon.noname-a con 0x7f812c0a9d70 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient(hunting): picked mon.noname-c con 0x7f812c13f650 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient(hunting): picked mon.noname-b con 0x7f812c1457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c141b60 con 0x7f812c0a9d70 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c140480 con 0x7f812c1457f0 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f812c140cd0 con 0x7f812c13f650 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:08.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient(hunting): authenticate will time out at 897.670121s 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8122ffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f812c13f650 0x7f812c1421f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:38532/0 (socket says 172.21.15.145:38532) 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f812b7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f812c0a9d70 0x7f812c144dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36510/0 (socket says 172.21.15.145:36510) 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8122ffd640 1 -- 172.21.15.145:0/3297109909 learned_addr learned my addr 172.21.15.145:0/3297109909 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 client.?.objecter ms_handle_connect 0x7f812c13f650 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 client.?.objecter ms_handle_connect 0x7f812c0a9d70 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087518125 0 0) 0x7f812c140cd0 con 0x7f812c13f650 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): my global_id is 4196 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8110004eb0 con 0x7f812c13f650 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 client.?.objecter ms_handle_connect 0x7f812c1457f0 2024-03-29T11:35:08.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180877618 0 0) 0x7f812c141b60 con 0x7f812c0a9d70 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812c140cd0 con 0x7f812c0a9d70 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048726650 0 0) 0x7f812c140480 con 0x7f812c1457f0 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f812c141b60 con 0x7f812c1457f0 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2720687772 0 0) 0x7f8110004eb0 con 0x7f812c13f650 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f812c140480 con 0x7f812c13f650 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f811c0013b0 con 0x7f812c13f650 2024-03-29T11:35:08.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 353444136 0 0) 0x7f812c140cd0 con 0x7f812c0a9d70 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8110004eb0 con 0x7f812c0a9d70 2024-03-29T11:35:08.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f81140017d0 con 0x7f812c0a9d70 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4093856990 0 0) 0x7f812c141b60 con 0x7f812c1457f0 2024-03-29T11:35:08.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f812c140cd0 con 0x7f812c1457f0 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8124001790 con 0x7f812c1457f0 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient(hunting): dump: 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:08.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2423746242 0 0) 0x7f812c140480 con 0x7f812c13f650 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 >> v1:172.21.15.203:6789/0 conn(0x7f812c1457f0 legacy=0x7f812c1e1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 >> v1:172.21.15.145:6789/0 conn(0x7f812c0a9d70 legacy=0x7f812c144dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 monclient: found mon.c 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c13dff0 con 0x7f812c13f650 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient: _finish_auth 0 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 5 monclient: authenticate success, global_id 4196 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient: _renew_subs 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812c13e2a0 con 0x7f812c13f650 2024-03-29T11:35:08.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:08.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient: _renew_subs 2024-03-29T11:35:08.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:08.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f812c142ae0 con 0x7f812c13f650 2024-03-29T11:35:08.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8131f0e640 1 librados: init done 2024-03-29T11:35:08.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f811c001610 con 0x7f812c13f650 2024-03-29T11:35:08.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f811c004c30 con 0x7f812c13f650 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.362+0000 7f8128ff9640 10 monclient: dump: 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:08.380 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:08.381 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:08.381 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:08.381 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:08.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f812a7fc640 4 set_mon_vals no callback set 2024-03-29T11:35:08.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f811c005120 con 0x7f812c13f650 2024-03-29T11:35:08.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f811c006990 con 0x7f812c13f650 2024-03-29T11:35:08.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8128ff9640 10 client.4196.objecter ms_dispatch 0x7f812c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:08.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8128ff9640 3 client.4196.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:08.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8128ff9640 3 client.4196.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:08.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8128ff9640 20 client.4196.objecter dump_active .. 0 homeless 2024-03-29T11:35:08.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8131f0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:08.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8131f0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:08.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8131f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:08.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.363+0000 7f8131f0e640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8104005080 con 0x7f812c13f650 2024-03-29T11:35:08.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.365+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f811c0062c0 con 0x7f812c13f650 2024-03-29T11:35:08.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.365+0000 7f8128ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:08.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.365+0000 7f8128ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:08.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.429+0000 7f8131f0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:08.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.429+0000 7f8131f0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:08.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.429+0000 7f8131f0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:08.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.429+0000 7f8131f0e640 1 -- 172.21.15.145:0/3297109909 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8104005330 con 0x7f812c13f650 2024-03-29T11:35:08.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.430+0000 7f8128ff9640 1 -- 172.21.15.145:0/3297109909 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f811c006570 con 0x7f812c13f650 2024-03-29T11:35:08.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.430+0000 7f8128ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:08.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.430+0000 7f8128ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:08.428 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:35:08.429 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 10 librados: watch_flush enter 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 10 librados: watch_flush exit 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 20 client.4196.objecter shutdown clearing up homeless session... 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 10 client.4196.objecter successfully canceled tick 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 10 monclient: shutdown 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 1 -- 172.21.15.145:0/3297109909 >> v1:172.21.15.145:6790/0 conn(0x7f812c13f650 legacy=0x7f812c1421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 1 -- 172.21.15.145:0/3297109909 shutdown_connections 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 1 -- 172.21.15.145:0/3297109909 >> 172.21.15.145:0/3297109909 conn(0x7f812c066a00 msgr2=0x7f812c0b4820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:08.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 1 -- 172.21.15.145:0/3297109909 shutdown_connections 2024-03-29T11:35:08.431 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 1 -- 172.21.15.145:0/3297109909 wait complete. 2024-03-29T11:35:08.431 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:08.431+0000 7f8131f0e640 1 librados: shutdown 2024-03-29T11:35:08.439 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:08.687 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:08.689+0000 7fe599e1e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:35:08.763 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:08.765+0000 7fa732632040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:35:08.961 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:08.963+0000 7fe599e1e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:35:09.013 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:09.015+0000 7fe599e1e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:35:09.039 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:09.041+0000 7fa732632040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:35:09.089 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:09.091+0000 7fa732632040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:35:09.127 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:09.129+0000 7fe599e1e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:35:09.212 INFO:tasks.ceph.mgr.x.smithi203.stderr:2024-03-29T11:35:09.214+0000 7fa732632040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:35:10.141 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:10.144+0000 7fe53a7dc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:35:10.142 INFO:tasks.ceph.mgr.y.smithi145.stderr:2024-03-29T11:35:10.144+0000 7fe53a7dc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:35:11.307 INFO:tasks.ceph.osd.2.smithi145.stderr:2024-03-29T11:35:11.310+0000 7fb528015640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:35:11.320 INFO:tasks.ceph.osd.6.smithi203.stderr:2024-03-29T11:35:11.322+0000 7f719e1f8640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:35:11.359 INFO:tasks.ceph.osd.4.smithi203.stderr:2024-03-29T11:35:11.361+0000 7efee03bd640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:35:11.365 INFO:tasks.ceph.osd.1.smithi145.stderr:2024-03-29T11:35:11.367+0000 7f5429214640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:35:11.366 INFO:tasks.ceph.osd.0.smithi145.stderr:2024-03-29T11:35:11.369+0000 7fb7728bd640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:35:11.372 INFO:tasks.ceph.osd.7.smithi203.stderr:2024-03-29T11:35:11.374+0000 7f005dc1e640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:35:11.381 INFO:tasks.ceph.osd.3.smithi145.stderr:2024-03-29T11:35:11.384+0000 7f89c1811640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:35:11.395 INFO:tasks.ceph.osd.5.smithi203.stderr:2024-03-29T11:35:11.398+0000 7f94aba17640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:35:15.246 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:15.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.405+0000 7feaad0d3640 10 monclient: get_monmap_and_config 2024-03-29T11:35:15.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.405+0000 7feaad0d3640 10 monclient: build_initial_monmap 2024-03-29T11:35:15.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.405+0000 7feaad0d3640 10 monclient: monmap: 2024-03-29T11:35:15.402 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:15.403 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:15.403 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:35:15.405480+0000 2024-03-29T11:35:15.403 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:35:15.405480+0000 2024-03-29T11:35:15.403 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:15.403 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:15.403 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:15.403 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:15.403 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:15.403 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:15.404 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.406+0000 7feaad0d3640 1 Processor -- start 2024-03-29T11:35:15.404 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.407+0000 7feaad0d3640 1 -- start start 2024-03-29T11:35:15.404 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.407+0000 7feaad0d3640 10 monclient: init 2024-03-29T11:35:15.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:15.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 10 monclient(hunting): picked mon.noname-c con 0x7feaa81457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 10 monclient(hunting): picked mon.noname-b con 0x7feaa813f650 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 10 monclient(hunting): picked mon.noname-a con 0x7feaa80a9d70 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa80aaef0 con 0x7feaa80a9d70 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa8145bd0 con 0x7feaa81457f0 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa8145eb0 con 0x7feaa813f650 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaad0d3640 10 monclient(hunting): authenticate will time out at 904.715700s 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaa6d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7feaa81457f0 0x7feaa813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:39396/0 (socket says 172.21.15.145:39396) 2024-03-29T11:35:15.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.408+0000 7feaa6d9d640 1 -- 172.21.15.145:0/3168005970 learned_addr learned my addr 172.21.15.145:0/3168005970 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309033602 0 0) 0x7feaa8145bd0 con 0x7feaa81457f0 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea90003520 con 0x7feaa81457f0 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719138452 0 0) 0x7feaa8145eb0 con 0x7feaa813f650 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:35:15.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.409+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea90003840 con 0x7feaa813f650 2024-03-29T11:35:15.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3697972628 0 0) 0x7fea90003520 con 0x7feaa81457f0 2024-03-29T11:35:15.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 >> v1:172.21.15.145:6790/0 conn(0x7feaa813f650 legacy=0x7feaa80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 >> v1:172.21.15.145:6789/0 conn(0x7feaa80a9d70 legacy=0x7feaa814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 1 monclient: found mon.noname-c 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa8146340 con 0x7feaa81457f0 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7feaa00019f0 con 0x7feaa81457f0 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: dump: 2024-03-29T11:35:15.409 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaad0d3640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaad0d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feaa0001b50 con 0x7feaa81457f0 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:15.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 1 -- 172.21.15.145:0/3168005970 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7feaa0001f60 con 0x7feaa81457f0 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaad0d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaad0d3640 4 set_mon_vals no callback set 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.410+0000 7feaa5d9b640 10 monclient: dump: 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:15.411 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:15.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.411+0000 7feaad0d3640 10 monclient: shutdown 2024-03-29T11:35:15.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.411+0000 7feaad0d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:15.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.411+0000 7feaad0d3640 1 -- 172.21.15.145:0/3168005970 >> v1:172.21.15.203:6789/0 conn(0x7feaa81457f0 legacy=0x7feaa813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.411+0000 7feaad0d3640 1 -- 172.21.15.145:0/3168005970 shutdown_connections 2024-03-29T11:35:15.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.411+0000 7feaad0d3640 1 -- 172.21.15.145:0/3168005970 >> 172.21.15.145:0/3168005970 conn(0x7feaa8066a00 msgr2=0x7feaa8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:15.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.411+0000 7feaad0d3640 1 -- 172.21.15.145:0/3168005970 shutdown_connections 2024-03-29T11:35:15.418 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.411+0000 7feaad0d3640 1 -- 172.21.15.145:0/3168005970 wait complete. 2024-03-29T11:35:15.418 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.412+0000 7feaad0d3640 10 monclient: build_initial_monmap 2024-03-29T11:35:15.418 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.412+0000 7feaad0d3640 10 monclient: monmap: 2024-03-29T11:35:15.418 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:15.418 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:15.418 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:15.418 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:35:15.418 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:15.418 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.413+0000 7feaad0d3640 1 librados: starting msgr at 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.413+0000 7feaad0d3640 1 librados: starting objecter 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.413+0000 7feaad0d3640 1 Processor -- start 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.414+0000 7feaad0d3640 1 -- start start 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.414+0000 7feaad0d3640 1 librados: setting wanted keys 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.414+0000 7feaad0d3640 1 librados: calling monclient init 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.414+0000 7feaad0d3640 10 monclient: init 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.414+0000 7feaad0d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.414+0000 7feaad0d3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:15.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.414+0000 7feaad0d3640 10 monclient(hunting): picked mon.noname-c con 0x7feaa80a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.414+0000 7feaad0d3640 10 monclient(hunting): picked mon.noname-a con 0x7feaa813f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7feaad0d3640 10 monclient(hunting): picked mon.noname-b con 0x7feaa81457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7feaad0d3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa8138820 con 0x7feaa813f650 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7feaad0d3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa806df30 con 0x7feaa81457f0 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7feaad0d3640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaa8072050 con 0x7feaa80a9d70 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7feaad0d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7feaad0d3640 10 monclient(hunting): authenticate will time out at 904.722232s 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7feaa759e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7feaa81457f0 0x7feaa81df520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:39540/0 (socket says 172.21.15.145:39540) 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7feaa759e640 1 -- 172.21.15.145:0/2504885323 learned_addr learned my addr 172.21.15.145:0/2504885323 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7fea8f7fe640 10 client.?.objecter ms_handle_connect 0x7feaa80a9d70 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7fea8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7fea8f7fe640 10 client.?.objecter ms_handle_connect 0x7feaa81457f0 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.415+0000 7fea8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113634900 0 0) 0x7feaa806df30 con 0x7feaa81457f0 2024-03-29T11:35:15.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea94004eb0 con 0x7feaa81457f0 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904503023 0 0) 0x7feaa8072050 con 0x7feaa80a9d70 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa806df30 con 0x7feaa80a9d70 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 client.?.objecter ms_handle_connect 0x7feaa813f650 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.416+0000 7fea8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 759189538 0 0) 0x7fea94004eb0 con 0x7feaa81457f0 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feaa8072050 con 0x7feaa81457f0 2024-03-29T11:35:15.421 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972265560 0 0) 0x7feaa8138820 con 0x7feaa813f650 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea94004eb0 con 0x7feaa813f650 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fea98001890 con 0x7feaa81457f0 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient(hunting): dump: 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:15.422 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1498358486 0 0) 0x7feaa8072050 con 0x7feaa81457f0 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 >> v1:172.21.15.145:6790/0 conn(0x7feaa80a9d70 legacy=0x7feaa8138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 >> v1:172.21.15.145:6789/0 conn(0x7feaa813f650 legacy=0x7feaa81dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 monclient: found mon.b 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.417+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaa81dfc30 con 0x7feaa81457f0 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7fea8f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:15.423 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7fea8f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7fea8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaad0d3640 5 monclient: authenticate success, global_id 4237 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaad0d3640 10 monclient: _renew_subs 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaad0d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa813ff10 con 0x7feaa81457f0 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaad0d3640 10 client.4237.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaad0d3640 10 monclient: _renew_subs 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaad0d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa806e710 con 0x7feaa81457f0 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaad0d3640 1 librados: init done 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea980019f0 con 0x7feaa81457f0 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7fea8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7feaa4d99640 4 set_mon_vals no callback set 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fea98003830 con 0x7feaa81457f0 2024-03-29T11:35:15.424 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7fea8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7fea8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.418+0000 7fea8f7fe640 10 monclient: dump: 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.419+0000 7feaad0d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.419+0000 7feaad0d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.419+0000 7feaad0d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:15.425 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.419+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea88005080 con 0x7feaa81457f0 2024-03-29T11:35:15.426 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.421+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fea98003a10 con 0x7feaa81457f0 2024-03-29T11:35:15.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.422+0000 7feaa6d9d640 10 monclient: get_auth_request con 0x7fea9406def0 auth_method 0 2024-03-29T11:35:15.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.422+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fea980848c0 con 0x7feaa81457f0 2024-03-29T11:35:15.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.422+0000 7fea8f7fe640 10 client.4237.objecter ms_dispatch 0x7feaa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:15.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.422+0000 7fea8f7fe640 3 client.4237.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:15.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.422+0000 7fea8f7fe640 3 client.4237.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:15.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.422+0000 7fea8f7fe640 20 client.4237.objecter dump_active .. 0 homeless 2024-03-29T11:35:15.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.422+0000 7fea8f7fe640 10 client.4237.objecter ms_handle_connect 0x7fea9406def0 2024-03-29T11:35:15.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.428+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea980501a0 con 0x7feaa81457f0 2024-03-29T11:35:15.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.428+0000 7fea8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:15.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.428+0000 7fea8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:15.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.814+0000 7feaad0d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:15.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.814+0000 7feaad0d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:15.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.814+0000 7feaad0d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:35:15.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.814+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea88005330 con 0x7feaa81457f0 2024-03-29T11:35:15.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.815+0000 7fea8f7fe640 1 -- 172.21.15.145:0/2504885323 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fea980520d0 con 0x7feaa81457f0 2024-03-29T11:35:15.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.815+0000 7fea8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:15.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.815+0000 7fea8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:15.813 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:35:15.813 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:15.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.819+0000 7feaad0d3640 10 librados: watch_flush enter 2024-03-29T11:35:15.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.819+0000 7feaad0d3640 10 librados: watch_flush exit 2024-03-29T11:35:15.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.819+0000 7feaad0d3640 20 client.4237.objecter shutdown clearing up homeless session... 2024-03-29T11:35:15.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.819+0000 7feaad0d3640 10 client.4237.objecter successfully canceled tick 2024-03-29T11:35:15.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.819+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 >> v1:172.21.15.145:6816/227372239 conn(0x7fea9406def0 legacy=0x7fea940703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.819+0000 7feaad0d3640 10 monclient: shutdown 2024-03-29T11:35:15.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.819+0000 7feaad0d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:15.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.819+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 >> v1:172.21.15.203:6789/0 conn(0x7feaa81457f0 legacy=0x7feaa81df520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:15.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.820+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 shutdown_connections 2024-03-29T11:35:15.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.820+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 >> 172.21.15.145:0/2504885323 conn(0x7feaa8066a00 msgr2=0x7feaa80b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:15.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.820+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 shutdown_connections 2024-03-29T11:35:15.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.820+0000 7feaad0d3640 1 -- 172.21.15.145:0/2504885323 wait complete. 2024-03-29T11:35:15.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:15.820+0000 7feaad0d3640 1 librados: shutdown 2024-03-29T11:35:15.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:22.639 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:22.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.782+0000 7fd81fc33640 10 monclient: get_monmap_and_config 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.782+0000 7fd81fc33640 10 monclient: build_initial_monmap 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.782+0000 7fd81fc33640 10 monclient: monmap: 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:35:22.783885+0000 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:35:22.783885+0000 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:22.781 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:22.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.783+0000 7fd81fc33640 1 Processor -- start 2024-03-29T11:35:22.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.783+0000 7fd81fc33640 1 -- start start 2024-03-29T11:35:22.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.783+0000 7fd81fc33640 10 monclient: init 2024-03-29T11:35:22.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:22.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:22.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 10 monclient(hunting): picked mon.noname-c con 0x7fd81806ebd0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:22.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 10 monclient(hunting): picked mon.noname-a con 0x7fd81806efb0 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:22.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 10 monclient(hunting): picked mon.noname-b con 0x7fd81813ed90 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd818070b40 con 0x7fd81806efb0 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd818070e20 con 0x7fd81806ebd0 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd818071100 con 0x7fd81813ed90 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81fc33640 10 monclient(hunting): authenticate will time out at 912.092568s 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81e1d0640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fd81813ed90 0x7fd818149d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:58388/0 (socket says 172.21.15.145:58388) 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.784+0000 7fd81e1d0640 1 -- 172.21.15.145:0/1599742315 learned_addr learned my addr 172.21.15.145:0/1599742315 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.785+0000 7fd81c9cd640 1 -- 172.21.15.145:0/1599742315 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49532239 0 0) 0x7fd818071100 con 0x7fd81813ed90 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.785+0000 7fd81c9cd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.785+0000 7fd81c9cd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.785+0000 7fd81c9cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.785+0000 7fd81c9cd640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.785+0000 7fd81c9cd640 1 -- 172.21.15.145:0/1599742315 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd804003520 con 0x7fd81813ed90 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.785+0000 7fd81c9cd640 1 -- 172.21.15.145:0/1599742315 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3178647893 0 0) 0x7fd804003520 con 0x7fd81813ed90 2024-03-29T11:35:22.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.785+0000 7fd81c9cd640 1 -- 172.21.15.145:0/1599742315 >> v1:172.21.15.203:6789/0 conn(0x7fd81806ebd0 legacy=0x7fd81813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:22.785 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.785+0000 7fd81c9cd640 1 -- 172.21.15.145:0/1599742315 >> v1:172.21.15.145:6789/0 conn(0x7fd81806efb0 legacy=0x7fd81813e680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 1 monclient: found mon.noname-b 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 1 -- 172.21.15.145:0/1599742315 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81813c040 con 0x7fd81813ed90 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 10 monclient: _finish_auth 0 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 1 -- 172.21.15.145:0/1599742315 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd810001a10 con 0x7fd81813ed90 2024-03-29T11:35:22.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81fc33640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81fc33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81c9cd640 10 monclient: dump: 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:22.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.787+0000 7fd81fc33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:22.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.788+0000 7fd81c9cd640 1 -- 172.21.15.145:0/1599742315 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd810001be0 con 0x7fd81813ed90 2024-03-29T11:35:22.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.788+0000 7fd81c9cd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:22.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.788+0000 7fd81fc33640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.788+0000 7fd81fc33640 4 set_mon_vals no callback set 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.788+0000 7fd81c9cd640 1 -- 172.21.15.145:0/1599742315 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd810002030 con 0x7fd81813ed90 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.788+0000 7fd81c9cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.788+0000 7fd81c9cd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.788+0000 7fd81c9cd640 10 monclient: dump: 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:22.789 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:22.790 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:22.790 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.789+0000 7fd81fc33640 10 monclient: shutdown 2024-03-29T11:35:22.790 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.789+0000 7fd81fc33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:22.790 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.789+0000 7fd81fc33640 1 -- 172.21.15.145:0/1599742315 >> v1:172.21.15.145:6790/0 conn(0x7fd81813ed90 legacy=0x7fd818149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:22.790 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.789+0000 7fd81fc33640 1 -- 172.21.15.145:0/1599742315 shutdown_connections 2024-03-29T11:35:22.790 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.789+0000 7fd81fc33640 1 -- 172.21.15.145:0/1599742315 >> 172.21.15.145:0/1599742315 conn(0x7fd8180b4200 msgr2=0x7fd8180b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:22.790 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.789+0000 7fd81fc33640 1 -- 172.21.15.145:0/1599742315 shutdown_connections 2024-03-29T11:35:22.790 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.789+0000 7fd81fc33640 1 -- 172.21.15.145:0/1599742315 wait complete. 2024-03-29T11:35:22.792 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.791+0000 7fd81fc33640 10 monclient: build_initial_monmap 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.791+0000 7fd81fc33640 10 monclient: monmap: 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.791+0000 7fd81fc33640 1 librados: starting msgr at 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.791+0000 7fd81fc33640 1 librados: starting objecter 2024-03-29T11:35:22.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.791+0000 7fd81fc33640 1 Processor -- start 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.791+0000 7fd81fc33640 1 -- start start 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.791+0000 7fd81fc33640 1 librados: setting wanted keys 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.791+0000 7fd81fc33640 1 librados: calling monclient init 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.791+0000 7fd81fc33640 10 monclient: init 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 10 monclient(hunting): picked mon.noname-c con 0x7fd81806ebd0 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 10 monclient(hunting): picked mon.noname-a con 0x7fd81806efb0 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 10 monclient(hunting): picked mon.noname-b con 0x7fd81813ed90 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd818141270 con 0x7fd81806efb0 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd818142410 con 0x7fd81813ed90 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd818140090 con 0x7fd81806ebd0 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81d9cf640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fd81806ebd0 0x7fd8181409a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:58670/0 (socket says 172.21.15.145:58670) 2024-03-29T11:35:22.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81d9cf640 1 -- 172.21.15.145:0/3716378994 learned_addr learned my addr 172.21.15.145:0/3716378994 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.792+0000 7fd81fc33640 10 monclient(hunting): authenticate will time out at 912.100928s 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 client.?.objecter ms_handle_connect 0x7fd81806ebd0 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430793863 0 0) 0x7fd818140090 con 0x7fd81806ebd0 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7fc004eb0 con 0x7fd81806ebd0 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1607672812 0 0) 0x7fd7fc004eb0 con 0x7fd81806ebd0 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd818140090 con 0x7fd81806ebd0 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd80c001960 con 0x7fd81806ebd0 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:22.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 monclient(hunting): dump: 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 client.?.objecter ms_handle_connect 0x7fd81813ed90 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.793+0000 7fd80a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.794+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3322089154 0 0) 0x7fd818140090 con 0x7fd81806ebd0 2024-03-29T11:35:22.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.794+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 >> v1:172.21.15.203:6789/0 conn(0x7fd81813ed90 legacy=0x7fd8181e5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:22.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.794+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 >> v1:172.21.15.145:6789/0 conn(0x7fd81806efb0 legacy=0x7fd8181db2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:22.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd80a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:22.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd80a7fc640 1 monclient: found mon.c 2024-03-29T11:35:22.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd80a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:22.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd80a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:22.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd818140220 con 0x7fd81806ebd0 2024-03-29T11:35:22.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd80a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:22.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd80a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:22.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd80a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:22.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd81fc33640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd81fc33640 10 monclient: _renew_subs 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd81fc33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd81fc33640 1 -- 172.21.15.145:0/3716378994 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8181404d0 con 0x7fd81806ebd0 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd81fc33640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd81fc33640 10 monclient: _renew_subs 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd81fc33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd81fc33640 1 -- 172.21.15.145:0/3716378994 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8181dbbc0 con 0x7fd81806ebd0 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.795+0000 7fd81fc33640 1 librados: init done 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.796+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd80c001c80 con 0x7fd81806ebd0 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.796+0000 7fd80a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.796+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd80c003a40 con 0x7fd81806ebd0 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.796+0000 7fd80a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.796+0000 7fd803fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.796+0000 7fd80a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:22.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.796+0000 7fd80a7fc640 10 monclient: dump: 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.797+0000 7fd803fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.797+0000 7fd803fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.797+0000 7fd803fff640 1 -- 172.21.15.145:0/3716378994 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd818067bf0 con 0x7fd81806ebd0 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.797+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fd80c003c60 con 0x7fd81806ebd0 2024-03-29T11:35:22.800 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.797+0000 7fd80bfff640 4 set_mon_vals no callback set 2024-03-29T11:35:22.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.798+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fd80c084ad0 con 0x7fd81806ebd0 2024-03-29T11:35:22.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.798+0000 7fd80a7fc640 10 client.4247.objecter ms_dispatch 0x7fd818000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:22.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.798+0000 7fd80a7fc640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:22.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.798+0000 7fd80a7fc640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:22.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.798+0000 7fd80a7fc640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-29T11:35:22.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.798+0000 7fd81d1ce640 10 monclient: get_auth_request con 0x7fd7fc06dcd0 auth_method 0 2024-03-29T11:35:22.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.798+0000 7fd80a7fc640 10 client.4247.objecter ms_handle_connect 0x7fd7fc06dcd0 2024-03-29T11:35:22.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.804+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd80c050410 con 0x7fd81806ebd0 2024-03-29T11:35:22.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.804+0000 7fd80a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:22.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:22.804+0000 7fd80a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:23.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.112+0000 7fd803fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:23.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.112+0000 7fd803fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:23.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.112+0000 7fd803fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:23.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.112+0000 7fd803fff640 1 -- 172.21.15.145:0/3716378994 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8181425a0 con 0x7fd81806ebd0 2024-03-29T11:35:23.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.113+0000 7fd80a7fc640 1 -- 172.21.15.145:0/3716378994 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fd8181425a0 con 0x7fd81806ebd0 2024-03-29T11:35:23.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.113+0000 7fd80a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:23.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.113+0000 7fd80a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:23.112 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:35:23.112 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 10 librados: watch_flush enter 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 10 librados: watch_flush exit 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 10 client.4247.objecter successfully canceled tick 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 1 -- 172.21.15.145:0/3716378994 >> v1:172.21.15.145:6816/227372239 conn(0x7fd7fc06dcd0 legacy=0x7fd7fc070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 10 monclient: shutdown 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 1 -- 172.21.15.145:0/3716378994 >> v1:172.21.15.145:6790/0 conn(0x7fd81806ebd0 legacy=0x7fd8181409a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 1 -- 172.21.15.145:0/3716378994 shutdown_connections 2024-03-29T11:35:23.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 1 -- 172.21.15.145:0/3716378994 >> 172.21.15.145:0/3716378994 conn(0x7fd8180b4200 msgr2=0x7fd818071c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:23.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 1 -- 172.21.15.145:0/3716378994 shutdown_connections 2024-03-29T11:35:23.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.116+0000 7fd803fff640 1 -- 172.21.15.145:0/3716378994 wait complete. 2024-03-29T11:35:23.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:23.117+0000 7fd803fff640 1 librados: shutdown 2024-03-29T11:35:23.128 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:29.935 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:30.079 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.080+0000 7f40f8e05640 10 monclient: get_monmap_and_config 2024-03-29T11:35:30.079 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.080+0000 7f40f8e05640 10 monclient: build_initial_monmap 2024-03-29T11:35:30.079 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.080+0000 7f40f8e05640 10 monclient: monmap: 2024-03-29T11:35:30.079 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:30.079 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:30.079 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:35:30.081964+0000 2024-03-29T11:35:30.080 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:35:30.081964+0000 2024-03-29T11:35:30.080 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:30.080 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:30.080 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:30.080 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:30.080 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:30.080 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:30.080 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.082+0000 7f40f8e05640 1 Processor -- start 2024-03-29T11:35:30.081 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.082+0000 7f40f8e05640 1 -- start start 2024-03-29T11:35:30.081 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.082+0000 7f40f8e05640 10 monclient: init 2024-03-29T11:35:30.081 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.083+0000 7f40f8e05640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:30.081 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.083+0000 7f40f8e05640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:35:30.081 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.083+0000 7f40f8e05640 10 monclient(hunting): picked mon.noname-a con 0x7f40f41422b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:30.087 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.084+0000 7f40f8e05640 10 monclient(hunting): picked mon.noname-b con 0x7f40f4142690 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:30.087 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.084+0000 7f40f8e05640 10 monclient(hunting): picked mon.noname-c con 0x7f40f40b17d0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.084+0000 7f40f8e05640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f4142c80 con 0x7f40f41422b0 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.084+0000 7f40f8e05640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f40aae40 con 0x7f40f40b17d0 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.084+0000 7f40f8e05640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f413fbd0 con 0x7f40f4142690 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.084+0000 7f40f8e05640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.084+0000 7f40f8e05640 10 monclient(hunting): authenticate will time out at 919.392935s 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.086+0000 7f40f259c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f40f41422b0 0x7f40f413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:50499/0 (socket says 172.21.15.145:50499) 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.086+0000 7f40f259c640 1 -- 172.21.15.145:0/2785316585 learned_addr learned my addr 172.21.15.145:0/2785316585 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 -- 172.21.15.145:0/2785316585 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824751076 0 0) 0x7f40f4142c80 con 0x7f40f41422b0 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 -- 172.21.15.145:0/2785316585 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40e0003520 con 0x7f40f41422b0 2024-03-29T11:35:30.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 -- 172.21.15.145:0/2785316585 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754829127 0 0) 0x7f40e0003520 con 0x7f40f41422b0 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 -- 172.21.15.145:0/2785316585 >> v1:172.21.15.145:6790/0 conn(0x7f40f4142690 legacy=0x7f40f40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 -- 172.21.15.145:0/2785316585 >> v1:172.21.15.203:6789/0 conn(0x7f40f40b17d0 legacy=0x7f40f40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 monclient: found mon.noname-a 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 -- 172.21.15.145:0/2785316585 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f4151940 con 0x7f40f41422b0 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: _finish_auth 0 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 -- 172.21.15.145:0/2785316585 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f40e4001690 con 0x7f40f41422b0 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: dump: 2024-03-29T11:35:30.089 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f8e05640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f8e05640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 -- 172.21.15.145:0/2785316585 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40e40018b0 con 0x7f40f41422b0 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:30.090 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 1 -- 172.21.15.145:0/2785316585 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f40e4001cc0 con 0x7f40f41422b0 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f1d9b640 10 monclient: dump: 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f8e05640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:30.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.087+0000 7f40f8e05640 4 set_mon_vals no callback set 2024-03-29T11:35:30.092 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.088+0000 7f40f8e05640 10 monclient: shutdown 2024-03-29T11:35:30.092 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.088+0000 7f40f8e05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:30.092 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.088+0000 7f40f8e05640 1 -- 172.21.15.145:0/2785316585 >> v1:172.21.15.145:6789/0 conn(0x7f40f41422b0 legacy=0x7f40f413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:30.092 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.089+0000 7f40f8e05640 1 -- 172.21.15.145:0/2785316585 shutdown_connections 2024-03-29T11:35:30.092 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.089+0000 7f40f8e05640 1 -- 172.21.15.145:0/2785316585 >> 172.21.15.145:0/2785316585 conn(0x7f40f4066a00 msgr2=0x7f40f4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.090+0000 7f40f8e05640 1 -- 172.21.15.145:0/2785316585 shutdown_connections 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.090+0000 7f40f8e05640 1 -- 172.21.15.145:0/2785316585 wait complete. 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.091+0000 7f40f8e05640 10 monclient: build_initial_monmap 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.091+0000 7f40f8e05640 10 monclient: monmap: 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:30.094 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.092+0000 7f40f8e05640 1 librados: starting msgr at 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.092+0000 7f40f8e05640 1 librados: starting objecter 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.092+0000 7f40f8e05640 1 Processor -- start 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.092+0000 7f40f8e05640 1 -- start start 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.092+0000 7f40f8e05640 1 librados: setting wanted keys 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.092+0000 7f40f8e05640 1 librados: calling monclient init 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.092+0000 7f40f8e05640 10 monclient: init 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.093+0000 7f40f8e05640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.093+0000 7f40f8e05640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.093+0000 7f40f8e05640 10 monclient(hunting): picked mon.noname-c con 0x7f40f40b17d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.093+0000 7f40f8e05640 10 monclient(hunting): picked mon.noname-b con 0x7f40f41422b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:30.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.094+0000 7f40ebfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f40f41422b0 0x7f40f412e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:36068/0 (socket says 172.21.15.145:36068) 2024-03-29T11:35:30.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40ebfff640 1 -- 172.21.15.145:0/2388311026 learned_addr learned my addr 172.21.15.145:0/2388311026 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:30.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40eaffd640 10 client.?.objecter ms_handle_connect 0x7f40f4142690 2024-03-29T11:35:30.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:30.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40f8e05640 10 monclient(hunting): picked mon.noname-a con 0x7f40f4142690 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:30.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f412f680 con 0x7f40f4142690 2024-03-29T11:35:30.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f4131780 con 0x7f40f41422b0 2024-03-29T11:35:30.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40f406eac0 con 0x7f40f40b17d0 2024-03-29T11:35:30.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40f8e05640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40f8e05640 10 monclient(hunting): authenticate will time out at 919.405454s 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495649211 0 0) 0x7f40f412f680 con 0x7f40f4142690 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40eaffd640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.097+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d4004eb0 con 0x7f40f4142690 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1716379648 0 0) 0x7f40d4004eb0 con 0x7f40f4142690 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40f412f680 con 0x7f40f4142690 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f40ec001b70 con 0x7f40f4142690 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 monclient(hunting): dump: 2024-03-29T11:35:30.099 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1928675 0 0) 0x7f40f412f680 con 0x7f40f4142690 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 >> v1:172.21.15.145:6790/0 conn(0x7f40f40b17d0 legacy=0x7f40f41322b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 >> v1:172.21.15.203:6789/0 conn(0x7f40f41422b0 legacy=0x7f40f412e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:30.100 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:30.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 1 monclient: found mon.a 2024-03-29T11:35:30.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:30.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:30.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f406eda0 con 0x7f40f4142690 2024-03-29T11:35:30.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:35:30.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:30.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:30.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 client.?.objecter ms_handle_connect 0x7f40f41422b0 2024-03-29T11:35:30.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.098+0000 7f40eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:30.103 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.100+0000 7f40f8e05640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:35:30.103 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.100+0000 7f40f8e05640 10 monclient: _renew_subs 2024-03-29T11:35:30.103 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.100+0000 7f40f8e05640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:30.103 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.100+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40f406f080 con 0x7f40f4142690 2024-03-29T11:35:30.103 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.100+0000 7f40f8e05640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:30.103 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.100+0000 7f40f8e05640 10 monclient: _renew_subs 2024-03-29T11:35:30.103 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.100+0000 7f40f8e05640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:30.103 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.100+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40f41329c0 con 0x7f40f4142690 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.100+0000 7f40f8e05640 1 librados: init done 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.101+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40ec001cd0 con 0x7f40f4142690 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.101+0000 7f40eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.101+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f40ec003b90 con 0x7f40f4142690 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.101+0000 7f40eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.101+0000 7f40eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.101+0000 7f40eaffd640 10 monclient: dump: 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:30.104 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:30.105 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:30.105 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:30.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.101+0000 7f40f159a640 4 set_mon_vals no callback set 2024-03-29T11:35:30.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.103+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f40ec019b60 con 0x7f40f4142690 2024-03-29T11:35:30.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.104+0000 7f40f8e05640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:30.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.104+0000 7f40f8e05640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:30.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.104+0000 7f40f8e05640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:30.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.104+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40f4075f10 con 0x7f40f4142690 2024-03-29T11:35:30.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.105+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f40ec083b70 con 0x7f40f4142690 2024-03-29T11:35:30.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.105+0000 7f40eaffd640 10 client.4248.objecter ms_dispatch 0x7f40f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:30.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.105+0000 7f40eaffd640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:30.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.105+0000 7f40eaffd640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:30.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.105+0000 7f40f259c640 10 monclient: get_auth_request con 0x7f40d406dc90 auth_method 0 2024-03-29T11:35:30.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.105+0000 7f40eaffd640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T11:35:30.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.106+0000 7f40eaffd640 10 client.4248.objecter ms_handle_connect 0x7f40d406dc90 2024-03-29T11:35:30.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.115+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40ec083e60 con 0x7f40f4142690 2024-03-29T11:35:30.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.115+0000 7f40eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:30.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.115+0000 7f40eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:30.431 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.433+0000 7f40f8e05640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:30.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.433+0000 7f40f8e05640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:30.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.433+0000 7f40f8e05640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:30.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.433+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40f40b2a30 con 0x7f40f4142690 2024-03-29T11:35:30.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.434+0000 7f40eaffd640 1 -- 172.21.15.145:0/2388311026 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f40ec04f450 con 0x7f40f4142690 2024-03-29T11:35:30.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.434+0000 7f40eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:30.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.434+0000 7f40eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:30.433 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:35:30.433 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:30.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.437+0000 7f40f8e05640 10 librados: watch_flush enter 2024-03-29T11:35:30.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.438+0000 7f40f8e05640 10 librados: watch_flush exit 2024-03-29T11:35:30.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.438+0000 7f40f8e05640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T11:35:30.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.438+0000 7f40f8e05640 10 client.4248.objecter successfully canceled tick 2024-03-29T11:35:30.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.438+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 >> v1:172.21.15.145:6816/227372239 conn(0x7f40d406dc90 legacy=0x7f40d4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:30.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.438+0000 7f40f8e05640 10 monclient: shutdown 2024-03-29T11:35:30.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.438+0000 7f40f8e05640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:30.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.438+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 >> v1:172.21.15.145:6789/0 conn(0x7f40f4142690 legacy=0x7f40f412ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:30.438 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.439+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 shutdown_connections 2024-03-29T11:35:30.438 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.440+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 >> 172.21.15.145:0/2388311026 conn(0x7f40f4066a00 msgr2=0x7f40f413c140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:30.438 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.440+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 shutdown_connections 2024-03-29T11:35:30.438 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.440+0000 7f40f8e05640 1 -- 172.21.15.145:0/2388311026 wait complete. 2024-03-29T11:35:30.438 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:30.440+0000 7f40f8e05640 1 librados: shutdown 2024-03-29T11:35:30.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:37.261 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:37.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.414+0000 7f85f4b92640 10 monclient: get_monmap_and_config 2024-03-29T11:35:37.417 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.414+0000 7f85f4b92640 10 monclient: build_initial_monmap 2024-03-29T11:35:37.419 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.415+0000 7f85f4b92640 10 monclient: monmap: 2024-03-29T11:35:37.419 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:37.419 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:37.419 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:35:37.416265+0000 2024-03-29T11:35:37.419 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:35:37.416265+0000 2024-03-29T11:35:37.419 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:37.419 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:37.419 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:37.420 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:37.420 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:37.420 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:37.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.419+0000 7f85f4b92640 1 Processor -- start 2024-03-29T11:35:37.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.419+0000 7f85f4b92640 1 -- start start 2024-03-29T11:35:37.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.419+0000 7f85f4b92640 10 monclient: init 2024-03-29T11:35:37.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.420+0000 7f85f4b92640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:37.420 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.420+0000 7f85f4b92640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:37.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.420+0000 7f85f4b92640 10 monclient(hunting): picked mon.noname-a con 0x7f85e8034850 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:37.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.421+0000 7f85f4b92640 10 monclient(hunting): picked mon.noname-c con 0x7f85e80c9550 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:37.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.421+0000 7f85f4b92640 10 monclient(hunting): picked mon.noname-b con 0x7f85e802aac0 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:37.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.421+0000 7f85f4b92640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e802c440 con 0x7f85e8034850 2024-03-29T11:35:37.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.421+0000 7f85f4b92640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e802aed0 con 0x7f85e80c9550 2024-03-29T11:35:37.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.421+0000 7f85f4b92640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e802b190 con 0x7f85e802aac0 2024-03-29T11:35:37.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.421+0000 7f85f4b92640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:37.422 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.421+0000 7f85f4b92640 10 monclient(hunting): authenticate will time out at 926.729540s 2024-03-29T11:35:37.425 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.427+0000 7f85effff640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f85e802aac0 0x7f85e80cc5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:39867/0 (socket says 172.21.15.145:39867) 2024-03-29T11:35:37.426 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.427+0000 7f85effff640 1 -- 172.21.15.145:0/223032719 learned_addr learned my addr 172.21.15.145:0/223032719 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:37.426 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.427+0000 7f85ee7fc640 1 -- 172.21.15.145:0/223032719 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236764243 0 0) 0x7f85e802b190 con 0x7f85e802aac0 2024-03-29T11:35:37.426 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.427+0000 7f85ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:35:37.426 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.427+0000 7f85ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:37.426 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.427+0000 7f85ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:37.426 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.427+0000 7f85ee7fc640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:35:37.426 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.427+0000 7f85ee7fc640 1 -- 172.21.15.145:0/223032719 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e0003520 con 0x7f85e802aac0 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 1 -- 172.21.15.145:0/223032719 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 147340727 0 0) 0x7f85e0003520 con 0x7f85e802aac0 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 1 -- 172.21.15.145:0/223032719 >> v1:172.21.15.203:6789/0 conn(0x7f85e80c9550 legacy=0x7f85e80cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 1 -- 172.21.15.145:0/223032719 >> v1:172.21.15.145:6789/0 conn(0x7f85e8034850 legacy=0x7f85e80c8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 1 monclient: found mon.noname-b 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 1 -- 172.21.15.145:0/223032719 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e8033060 con 0x7f85e802aac0 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:35:37.427 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85f4b92640 5 monclient: authenticate success, global_id 4253 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85f4b92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 1 -- 172.21.15.145:0/223032719 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f85e4001690 con 0x7f85e802aac0 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 10 monclient: dump: 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:37.428 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:37.429 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:37.429 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85f4b92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:37.429 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 1 -- 172.21.15.145:0/223032719 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85e4001920 con 0x7f85e802aac0 2024-03-29T11:35:37.429 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:37.429 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85f4b92640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:37.429 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.428+0000 7f85f4b92640 4 set_mon_vals no callback set 2024-03-29T11:35:37.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.429+0000 7f85f4b92640 10 monclient: shutdown 2024-03-29T11:35:37.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.429+0000 7f85f4b92640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:37.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.429+0000 7f85f4b92640 1 -- 172.21.15.145:0/223032719 >> v1:172.21.15.145:6790/0 conn(0x7f85e802aac0 legacy=0x7f85e80cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.429+0000 7f85f4b92640 1 -- 172.21.15.145:0/223032719 shutdown_connections 2024-03-29T11:35:37.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.429+0000 7f85f4b92640 1 -- 172.21.15.145:0/223032719 >> 172.21.15.145:0/223032719 conn(0x7f85e80058f0 msgr2=0x7f85e8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:37.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.429+0000 7f85f4b92640 1 -- 172.21.15.145:0/223032719 shutdown_connections 2024-03-29T11:35:37.430 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.430+0000 7f85f4b92640 1 -- 172.21.15.145:0/223032719 wait complete. 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.431+0000 7f85f4b92640 10 monclient: build_initial_monmap 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.431+0000 7f85f4b92640 10 monclient: monmap: 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.431+0000 7f85f4b92640 1 librados: starting msgr at 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.431+0000 7f85f4b92640 1 librados: starting objecter 2024-03-29T11:35:37.431 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.431+0000 7f85f4b92640 1 Processor -- start 2024-03-29T11:35:37.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.432+0000 7f85f4b92640 1 -- start start 2024-03-29T11:35:37.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.432+0000 7f85f4b92640 1 librados: setting wanted keys 2024-03-29T11:35:37.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.432+0000 7f85f4b92640 1 librados: calling monclient init 2024-03-29T11:35:37.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.432+0000 7f85f4b92640 10 monclient: init 2024-03-29T11:35:37.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.432+0000 7f85f4b92640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:37.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.432+0000 7f85f4b92640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:35:37.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.433+0000 7f85f4b92640 10 monclient(hunting): picked mon.noname-b con 0x7f85e8034850 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:37.432 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.433+0000 7f85f4b92640 10 monclient(hunting): picked mon.noname-c con 0x7f85e80c9550 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:37.433 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.433+0000 7f85f4b92640 10 monclient(hunting): picked mon.noname-a con 0x7f85e81649c0 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:37.433 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.433+0000 7f85f4b92640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e802dc10 con 0x7f85e81649c0 2024-03-29T11:35:37.433 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.433+0000 7f85f4b92640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e802deb0 con 0x7f85e8034850 2024-03-29T11:35:37.433 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.433+0000 7f85f4b92640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85e802ec50 con 0x7f85e80c9550 2024-03-29T11:35:37.433 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.433+0000 7f85f4b92640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:37.433 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.433+0000 7f85f4b92640 10 monclient(hunting): authenticate will time out at 926.741106s 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85eeffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f85e80c9550 0x7f85e80c2b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:39943/0 (socket says 172.21.15.145:39943) 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85eeffd640 1 -- 172.21.15.145:0/452672501 learned_addr learned my addr 172.21.15.145:0/452672501 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 10 client.?.objecter ms_handle_connect 0x7f85e80c9550 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949654177 0 0) 0x7f85e802ec50 con 0x7f85e80c9550 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d4004eb0 con 0x7f85e80c9550 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 10 client.?.objecter ms_handle_connect 0x7f85e81649c0 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.437+0000 7f85dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:37.441 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027837727 0 0) 0x7f85e802dc10 con 0x7f85e81649c0 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e802ec50 con 0x7f85e81649c0 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2783393898 0 0) 0x7f85d4004eb0 con 0x7f85e80c9550 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85e802dc10 con 0x7f85e80c9550 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f85dc0017d0 con 0x7f85e80c9550 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 10 monclient(hunting): dump: 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:37.442 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.438+0000 7f85dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 10 client.?.objecter ms_handle_connect 0x7f85e8034850 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1131398325 0 0) 0x7f85e802ec50 con 0x7f85e81649c0 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85d4004eb0 con 0x7f85e81649c0 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785870473 0 0) 0x7f85e802deb0 con 0x7f85e8034850 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:37.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85e802ec50 con 0x7f85e8034850 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3270731536 0 0) 0x7f85e802dc10 con 0x7f85e80c9550 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 >> v1:172.21.15.203:6789/0 conn(0x7f85e8034850 legacy=0x7f85e80bf6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 >> v1:172.21.15.145:6789/0 conn(0x7f85e81649c0 legacy=0x7f85e80c3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 1 monclient: found mon.c 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.439+0000 7f85dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e802ef30 con 0x7f85e80c9550 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 10 monclient: _finish_auth 0 2024-03-29T11:35:37.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85dc001ce0 con 0x7f85e80c9550 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f85dc004a00 con 0x7f85e80c9550 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85dbfff640 10 monclient: dump: 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:37.445 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85f4b92640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85f4b92640 10 monclient: _renew_subs 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85f4b92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85e80bfdf0 con 0x7f85e80c9550 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85f4b92640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:37.446 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85f4b92640 10 monclient: _renew_subs 2024-03-29T11:35:37.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85f4b92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:37.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85e8164160 con 0x7f85e80c9550 2024-03-29T11:35:37.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.440+0000 7f85f4b92640 1 librados: init done 2024-03-29T11:35:37.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.442+0000 7f85ed7fa640 4 set_mon_vals no callback set 2024-03-29T11:35:37.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.442+0000 7f85f4b92640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.443+0000 7f85f4b92640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.443+0000 7f85f4b92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.443+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85e80c0140 con 0x7f85e80c9550 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.444+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f85dc01a9d0 con 0x7f85e80c9550 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.444+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f85dc0849b0 con 0x7f85e80c9550 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.445+0000 7f85dbfff640 10 client.4256.objecter ms_dispatch 0x7f85f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.445+0000 7f85dbfff640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.445+0000 7f85dbfff640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.445+0000 7f85dbfff640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.446+0000 7f85ef7fe640 10 monclient: get_auth_request con 0x7f85d406dd10 auth_method 0 2024-03-29T11:35:37.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.446+0000 7f85dbfff640 10 client.4256.objecter ms_handle_connect 0x7f85d406dd10 2024-03-29T11:35:37.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.454+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85dc050290 con 0x7f85e80c9550 2024-03-29T11:35:37.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.455+0000 7f85dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:37.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.455+0000 7f85dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:37.772 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.773+0000 7f85f4b92640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:37.772 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.773+0000 7f85f4b92640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:37.772 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.773+0000 7f85f4b92640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:37.772 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.773+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85e8004cd0 con 0x7f85e80c9550 2024-03-29T11:35:37.775 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.776+0000 7f85dbfff640 1 -- 172.21.15.145:0/452672501 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f85dc0521c0 con 0x7f85e80c9550 2024-03-29T11:35:37.775 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.776+0000 7f85dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:37.775 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.776+0000 7f85dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:37.775 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:35:37.775 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:37.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.780+0000 7f85f4b92640 10 librados: watch_flush enter 2024-03-29T11:35:37.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.780+0000 7f85f4b92640 10 librados: watch_flush exit 2024-03-29T11:35:37.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.780+0000 7f85f4b92640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T11:35:37.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.780+0000 7f85f4b92640 10 client.4256.objecter successfully canceled tick 2024-03-29T11:35:37.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.780+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 >> v1:172.21.15.145:6816/227372239 conn(0x7f85d406dd10 legacy=0x7f85d40701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.780+0000 7f85f4b92640 10 monclient: shutdown 2024-03-29T11:35:37.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.780+0000 7f85f4b92640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:37.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.780+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 >> v1:172.21.15.145:6790/0 conn(0x7f85e80c9550 legacy=0x7f85e80c2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:37.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.781+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 shutdown_connections 2024-03-29T11:35:37.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.781+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 >> 172.21.15.145:0/452672501 conn(0x7f85e80058f0 msgr2=0x7f85e80c76c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:37.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.781+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 shutdown_connections 2024-03-29T11:35:37.781 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.782+0000 7f85f4b92640 1 -- 172.21.15.145:0/452672501 wait complete. 2024-03-29T11:35:37.781 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:37.782+0000 7f85f4b92640 1 librados: shutdown 2024-03-29T11:35:37.798 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:44.605 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:44.750 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.751+0000 7fef54b25640 10 monclient: get_monmap_and_config 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.751+0000 7fef54b25640 10 monclient: build_initial_monmap 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.751+0000 7fef54b25640 10 monclient: monmap: 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:35:44.753127+0000 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:35:44.753127+0000 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:44.751 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:44.753 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.755+0000 7fef54b25640 1 Processor -- start 2024-03-29T11:35:44.753 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.755+0000 7fef54b25640 1 -- start start 2024-03-29T11:35:44.753 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.755+0000 7fef54b25640 10 monclient: init 2024-03-29T11:35:44.758 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:44.758 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 10 monclient(hunting): picked mon.noname-b con 0x7fef501457f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 10 monclient(hunting): picked mon.noname-c con 0x7fef5013f650 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 10 monclient(hunting): picked mon.noname-a con 0x7fef500a9d70 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef500aaef0 con 0x7fef500a9d70 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef50145bd0 con 0x7fef5013f650 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef50145eb0 con 0x7fef501457f0 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.756+0000 7fef54b25640 10 monclient(hunting): authenticate will time out at 934.064431s 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.760+0000 7fef4ed9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fef500a9d70 0x7fef5014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:41475/0 (socket says 172.21.15.145:41475) 2024-03-29T11:35:44.761 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.760+0000 7fef4ed9d640 1 -- 172.21.15.145:0/2578110254 learned_addr learned my addr 172.21.15.145:0/2578110254 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:44.762 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.763+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405823245 0 0) 0x7fef50145bd0 con 0x7fef5013f650 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef38003520 con 0x7fef5013f650 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342197924 0 0) 0x7fef500aaef0 con 0x7fef500a9d70 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef50145bd0 con 0x7fef500a9d70 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865097739 0 0) 0x7fef50145eb0 con 0x7fef501457f0 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:44.763 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef500aaef0 con 0x7fef501457f0 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1122114474 0 0) 0x7fef38003520 con 0x7fef5013f650 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 >> v1:172.21.15.145:6790/0 conn(0x7fef501457f0 legacy=0x7fef5013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 >> v1:172.21.15.145:6789/0 conn(0x7fef500a9d70 legacy=0x7fef5014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 monclient: found mon.noname-c 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef50146340 con 0x7fef5013f650 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient: _finish_auth 0 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:44.764 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fef48001940 con 0x7fef5013f650 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.764+0000 7fef4d59a640 10 monclient: dump: 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:44.765 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:44.767 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.766+0000 7fef54b25640 5 monclient: authenticate success, global_id 4249 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.766+0000 7fef54b25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.766+0000 7fef4d59a640 1 -- 172.21.15.145:0/2578110254 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef48001aa0 con 0x7fef5013f650 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.766+0000 7fef4d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.766+0000 7fef54b25640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.766+0000 7fef54b25640 4 set_mon_vals no callback set 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.767+0000 7fef54b25640 10 monclient: shutdown 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.767+0000 7fef54b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.767+0000 7fef54b25640 1 -- 172.21.15.145:0/2578110254 >> v1:172.21.15.203:6789/0 conn(0x7fef5013f650 legacy=0x7fef500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.767+0000 7fef54b25640 1 -- 172.21.15.145:0/2578110254 shutdown_connections 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.767+0000 7fef54b25640 1 -- 172.21.15.145:0/2578110254 >> 172.21.15.145:0/2578110254 conn(0x7fef50066a00 msgr2=0x7fef50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.767+0000 7fef54b25640 1 -- 172.21.15.145:0/2578110254 shutdown_connections 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.767+0000 7fef54b25640 1 -- 172.21.15.145:0/2578110254 wait complete. 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.768+0000 7fef54b25640 10 monclient: build_initial_monmap 2024-03-29T11:35:44.768 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.768+0000 7fef54b25640 10 monclient: monmap: 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.768+0000 7fef54b25640 1 librados: starting msgr at 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.768+0000 7fef54b25640 1 librados: starting objecter 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.768+0000 7fef54b25640 1 Processor -- start 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.769+0000 7fef54b25640 1 -- start start 2024-03-29T11:35:44.769 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.769+0000 7fef54b25640 1 librados: setting wanted keys 2024-03-29T11:35:44.770 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.769+0000 7fef54b25640 1 librados: calling monclient init 2024-03-29T11:35:44.770 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.769+0000 7fef54b25640 10 monclient: init 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.769+0000 7fef54b25640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.770+0000 7fef54b25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.770+0000 7fef54b25640 10 monclient(hunting): picked mon.noname-a con 0x7fef500a9d70 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.770+0000 7fef54b25640 10 monclient(hunting): picked mon.noname-b con 0x7fef501457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.770+0000 7fef54b25640 10 monclient(hunting): picked mon.noname-c con 0x7fef501384e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.770+0000 7fef54b25640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5006fcc0 con 0x7fef500a9d70 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.770+0000 7fef54b25640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef5006eae0 con 0x7fef501457f0 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.770+0000 7fef54b25640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef50138de0 con 0x7fef501384e0 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.770+0000 7fef54b25640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:44.771 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.770+0000 7fef54b25640 10 monclient(hunting): authenticate will time out at 934.078538s 2024-03-29T11:35:44.772 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.773+0000 7fef4ed9d640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fef501384e0 0x7fef501e6ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:45559/0 (socket says 172.21.15.145:45559) 2024-03-29T11:35:44.772 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.773+0000 7fef4ed9d640 1 -- 172.21.15.145:0/3423855626 learned_addr learned my addr 172.21.15.145:0/3423855626 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:44.781 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 client.?.objecter ms_handle_connect 0x7fef501457f0 2024-03-29T11:35:44.781 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:44.781 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345691624 0 0) 0x7fef5006eae0 con 0x7fef501457f0 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef30004eb0 con 0x7fef501457f0 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 client.?.objecter ms_handle_connect 0x7fef500a9d70 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818828273 0 0) 0x7fef5006fcc0 con 0x7fef500a9d70 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:35:44.782 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef5006eae0 con 0x7fef500a9d70 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 client.?.objecter ms_handle_connect 0x7fef501384e0 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.776+0000 7fef3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1710258151 0 0) 0x7fef5006eae0 con 0x7fef500a9d70 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef5006fcc0 con 0x7fef500a9d70 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3248217436 0 0) 0x7fef30004eb0 con 0x7fef501457f0 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef5006eae0 con 0x7fef501457f0 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350186595 0 0) 0x7fef50138de0 con 0x7fef501384e0 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): _negotiate 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef30004eb0 con 0x7fef501384e0 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fef44001740 con 0x7fef500a9d70 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:44.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): dump: 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fef48003b10 con 0x7fef501457f0 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:44.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.777+0000 7fef3effd640 10 monclient(hunting): dump: 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 888307015 0 0) 0x7fef5006fcc0 con 0x7fef500a9d70 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 >> v1:172.21.15.145:6790/0 conn(0x7fef501384e0 legacy=0x7fef501e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 >> v1:172.21.15.203:6789/0 conn(0x7fef501457f0 legacy=0x7fef50137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:44.785 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 10 monclient: _finish_hunting 0 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 1 monclient: found mon.a 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef50138f70 con 0x7fef500a9d70 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 10 monclient: _finish_auth 0 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 10 monclient: _check_auth_tickets 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef54b25640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef54b25640 10 monclient: _renew_subs 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef54b25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef501df860 con 0x7fef500a9d70 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef54b25640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef54b25640 10 monclient: _renew_subs 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef54b25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:44.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef5006e220 con 0x7fef500a9d70 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef54b25640 1 librados: init done 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef440018a0 con 0x7fef500a9d70 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fef440037b0 con 0x7fef500a9d70 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef3effd640 10 monclient: dump: 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:44.787 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.778+0000 7fef4cd99640 4 set_mon_vals no callback set 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.780+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fef44003910 con 0x7fef500a9d70 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.780+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fef440836f0 con 0x7fef500a9d70 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.780+0000 7fef3effd640 10 client.4260.objecter ms_dispatch 0x7fef50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.780+0000 7fef3effd640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.780+0000 7fef3effd640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.780+0000 7fef3effd640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.781+0000 7fef54b25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.782+0000 7fef54b25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.782+0000 7fef54b25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:44.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.782+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef34005080 con 0x7fef500a9d70 2024-03-29T11:35:44.789 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.784+0000 7fef4dd9b640 10 monclient: get_auth_request con 0x7fef3006dfa0 auth_method 0 2024-03-29T11:35:44.789 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.784+0000 7fef3effd640 10 client.4260.objecter ms_handle_connect 0x7fef3006dfa0 2024-03-29T11:35:44.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.793+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef4404ee00 con 0x7fef500a9d70 2024-03-29T11:35:44.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.793+0000 7fef3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:44.792 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:44.793+0000 7fef3effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:45.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.113+0000 7fef54b25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:45.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.113+0000 7fef54b25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:45.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.113+0000 7fef54b25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:45.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.113+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef34005330 con 0x7fef500a9d70 2024-03-29T11:35:45.114 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.116+0000 7fef3effd640 1 -- 172.21.15.145:0/3423855626 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fef44050f00 con 0x7fef500a9d70 2024-03-29T11:35:45.114 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.116+0000 7fef3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:45.114 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.116+0000 7fef3effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:45.119 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:35:45.119 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:45.121 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.120+0000 7fef54b25640 10 librados: watch_flush enter 2024-03-29T11:35:45.121 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.120+0000 7fef54b25640 10 librados: watch_flush exit 2024-03-29T11:35:45.121 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.120+0000 7fef54b25640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-29T11:35:45.121 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.120+0000 7fef54b25640 10 client.4260.objecter successfully canceled tick 2024-03-29T11:35:45.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.120+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 >> v1:172.21.15.145:6816/227372239 conn(0x7fef3006dfa0 legacy=0x7fef30070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:45.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.120+0000 7fef54b25640 10 monclient: shutdown 2024-03-29T11:35:45.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.120+0000 7fef54b25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:45.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.120+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 >> v1:172.21.15.145:6789/0 conn(0x7fef500a9d70 legacy=0x7fef501df150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:45.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.121+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 shutdown_connections 2024-03-29T11:35:45.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.121+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 >> 172.21.15.145:0/3423855626 conn(0x7fef50066a00 msgr2=0x7fef500b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:45.123 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.121+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 shutdown_connections 2024-03-29T11:35:45.123 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.122+0000 7fef54b25640 1 -- 172.21.15.145:0/3423855626 wait complete. 2024-03-29T11:35:45.124 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:45.124+0000 7fef54b25640 1 librados: shutdown 2024-03-29T11:35:45.141 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:51.948 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:52.100 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.101+0000 7f8f4209d640 10 monclient: get_monmap_and_config 2024-03-29T11:35:52.100 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.101+0000 7f8f4209d640 10 monclient: build_initial_monmap 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.101+0000 7f8f4209d640 10 monclient: monmap: 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:35:52.103158+0000 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:35:52.103158+0000 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:52.101 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:52.102 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.102+0000 7f8f4209d640 1 Processor -- start 2024-03-29T11:35:52.102 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.103+0000 7f8f4209d640 1 -- start start 2024-03-29T11:35:52.102 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.103+0000 7f8f4209d640 10 monclient: init 2024-03-29T11:35:52.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:52.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:52.104 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f34038c60 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:52.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f340324b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:52.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f340d11e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:52.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f340caf80 con 0x7f8f34038c60 2024-03-29T11:35:52.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f34039850 con 0x7f8f340324b0 2024-03-29T11:35:52.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f340c78d0 con 0x7f8f340d11e0 2024-03-29T11:35:52.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:52.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.104+0000 7f8f4209d640 10 monclient(hunting): authenticate will time out at 941.412310s 2024-03-29T11:35:52.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.109+0000 7f8f4109b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f8f34038c60 0x7f8f340c4ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:32893/0 (socket says 172.21.15.145:32893) 2024-03-29T11:35:52.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.109+0000 7f8f4109b640 1 -- 172.21.15.145:0/1132504544 learned_addr learned my addr 172.21.15.145:0/1132504544 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435319693 0 0) 0x7f8f340caf80 con 0x7f8f34038c60 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f28003520 con 0x7f8f34038c60 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239684083 0 0) 0x7f8f34039850 con 0x7f8f340324b0 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:52.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.110+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f340caf80 con 0x7f8f340324b0 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665731875 0 0) 0x7f8f28003520 con 0x7f8f34038c60 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 >> v1:172.21.15.145:6790/0 conn(0x7f8f340d11e0 legacy=0x7f8f340d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 >> v1:172.21.15.203:6789/0 conn(0x7f8f340324b0 legacy=0x7f8f340cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 10 monclient: _finish_hunting 0 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 1 monclient: found mon.noname-a 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f340c7a60 con 0x7f8f34038c60 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 10 monclient: _finish_auth 0 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 10 monclient: _check_auth_tickets 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8f2c001690 con 0x7f8f34038c60 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:52.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f33fff640 10 monclient: dump: 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f4209d640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:35:52.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.111+0000 7f8f4209d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:52.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.112+0000 7f8f33fff640 1 -- 172.21.15.145:0/1132504544 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f2c0017f0 con 0x7f8f34038c60 2024-03-29T11:35:52.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.112+0000 7f8f33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:52.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.112+0000 7f8f4209d640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:52.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.112+0000 7f8f4209d640 4 set_mon_vals no callback set 2024-03-29T11:35:52.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.113+0000 7f8f4209d640 10 monclient: shutdown 2024-03-29T11:35:52.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.113+0000 7f8f4209d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:52.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.113+0000 7f8f4209d640 1 -- 172.21.15.145:0/1132504544 >> v1:172.21.15.145:6789/0 conn(0x7f8f34038c60 legacy=0x7f8f340c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:52.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.113+0000 7f8f4209d640 1 -- 172.21.15.145:0/1132504544 shutdown_connections 2024-03-29T11:35:52.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.113+0000 7f8f4209d640 1 -- 172.21.15.145:0/1132504544 >> 172.21.15.145:0/1132504544 conn(0x7f8f34005c20 msgr2=0x7f8f34006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:52.114 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.114+0000 7f8f4209d640 1 -- 172.21.15.145:0/1132504544 shutdown_connections 2024-03-29T11:35:52.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.115+0000 7f8f4209d640 1 -- 172.21.15.145:0/1132504544 wait complete. 2024-03-29T11:35:52.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.116+0000 7f8f4209d640 10 monclient: build_initial_monmap 2024-03-29T11:35:52.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.116+0000 7f8f4209d640 10 monclient: monmap: 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.116+0000 7f8f4209d640 1 librados: starting msgr at 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.116+0000 7f8f4209d640 1 librados: starting objecter 2024-03-29T11:35:52.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.116+0000 7f8f4209d640 1 Processor -- start 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 1 -- start start 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 1 librados: setting wanted keys 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 1 librados: calling monclient init 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 10 monclient: init 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f340324b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f340d11e0 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f340c22a0 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f340c3220 con 0x7f8f340324b0 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3402eb20 con 0x7f8f340d11e0 2024-03-29T11:35:52.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f340c3490 con 0x7f8f340c22a0 2024-03-29T11:35:52.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:52.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.117+0000 7f8f4209d640 10 monclient(hunting): authenticate will time out at 941.425940s 2024-03-29T11:35:52.125 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.122+0000 7f8f4089a640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f8f340d11e0 0x7f8f340c3b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:53295/0 (socket says 172.21.15.145:53295) 2024-03-29T11:35:52.125 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.122+0000 7f8f4109b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f8f340324b0 0x7f8f3402fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:32985/0 (socket says 172.21.15.145:32985) 2024-03-29T11:35:52.125 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.122+0000 7f8f4109b640 1 -- 172.21.15.145:0/3550939389 learned_addr learned my addr 172.21.15.145:0/3550939389 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:52.125 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 client.?.objecter ms_handle_connect 0x7f8f340324b0 2024-03-29T11:35:52.125 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:52.125 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 client.?.objecter ms_handle_connect 0x7f8f340c22a0 2024-03-29T11:35:52.125 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:52.125 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1862706810 0 0) 0x7f8f340c3490 con 0x7f8f340c22a0 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f20004eb0 con 0x7f8f340c22a0 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 client.?.objecter ms_handle_connect 0x7f8f340d11e0 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109297926 0 0) 0x7f8f3402eb20 con 0x7f8f340d11e0 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f340c3490 con 0x7f8f340d11e0 2024-03-29T11:35:52.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2859089818 0 0) 0x7f8f20004eb0 con 0x7f8f340c22a0 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f20005490 con 0x7f8f340c22a0 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8f3c04fdd0 con 0x7f8f340c22a0 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): dump: 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:52.127 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.123+0000 7f8f31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047341148 0 0) 0x7f8f340c3220 con 0x7f8f340324b0 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f20004eb0 con 0x7f8f340324b0 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 744387589 0 0) 0x7f8f20005490 con 0x7f8f340c22a0 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 >> v1:172.21.15.203:6789/0 conn(0x7f8f340d11e0 legacy=0x7f8f340c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 >> v1:172.21.15.145:6789/0 conn(0x7f8f340324b0 legacy=0x7f8f3402fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 1 monclient: found mon.c 2024-03-29T11:35:52.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.124+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f3402f7a0 con 0x7f8f340c22a0 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.125+0000 7f8f31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.125+0000 7f8f31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.125+0000 7f8f31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.125+0000 7f8f4209d640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.125+0000 7f8f4209d640 10 monclient: _renew_subs 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.125+0000 7f8f4209d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.125+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f340c42a0 con 0x7f8f340c22a0 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.126+0000 7f8f4209d640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.126+0000 7f8f4209d640 10 monclient: _renew_subs 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.126+0000 7f8f4209d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.126+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f340c4730 con 0x7f8f340c22a0 2024-03-29T11:35:52.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.126+0000 7f8f4209d640 1 librados: init done 2024-03-29T11:35:52.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.129+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f3c04f400 con 0x7f8f340c22a0 2024-03-29T11:35:52.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.129+0000 7f8f31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:52.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.129+0000 7f8f4209d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:52.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.129+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8f3c04f730 con 0x7f8f340c22a0 2024-03-29T11:35:52.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.129+0000 7f8f31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:52.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.129+0000 7f8f31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:35:52.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.129+0000 7f8f31ffb640 10 monclient: dump: 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.129+0000 7f8f337fe640 4 set_mon_vals no callback set 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.129+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f8f3c075710 con 0x7f8f340c22a0 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.130+0000 7f8f4209d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.130+0000 7f8f4209d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:52.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.130+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f34007b90 con 0x7f8f340c22a0 2024-03-29T11:35:52.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.130+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f8f3c0fe090 con 0x7f8f340c22a0 2024-03-29T11:35:52.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.130+0000 7f8f31ffb640 10 client.4265.objecter ms_dispatch 0x7f8f3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:52.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.130+0000 7f8f31ffb640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:52.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.130+0000 7f8f31ffb640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:52.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.130+0000 7f8f31ffb640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:35:52.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.132+0000 7f8f4109b640 10 monclient: get_auth_request con 0x7f8f2006df10 auth_method 0 2024-03-29T11:35:52.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.133+0000 7f8f31ffb640 10 client.4265.objecter ms_handle_connect 0x7f8f2006df10 2024-03-29T11:35:52.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.138+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f3c0c99f0 con 0x7f8f340c22a0 2024-03-29T11:35:52.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.138+0000 7f8f31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:52.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.138+0000 7f8f31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:52.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.462+0000 7f8f4209d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:52.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.462+0000 7f8f4209d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:52.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.463+0000 7f8f4209d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:35:52.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.463+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f34004cd0 con 0x7f8f340c22a0 2024-03-29T11:35:52.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.464+0000 7f8f31ffb640 1 -- 172.21.15.145:0/3550939389 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f8f3c0cb920 con 0x7f8f340c22a0 2024-03-29T11:35:52.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.465+0000 7f8f31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:52.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.465+0000 7f8f31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:52.464 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:35:52.465 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:52.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.469+0000 7f8f4209d640 10 librados: watch_flush enter 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 10 librados: watch_flush exit 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 >> v1:172.21.15.145:6816/227372239 conn(0x7f8f2006df10 legacy=0x7f8f200703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 10 monclient: shutdown 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 >> v1:172.21.15.145:6790/0 conn(0x7f8f340c22a0 legacy=0x7f8f34173dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 shutdown_connections 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 >> 172.21.15.145:0/3550939389 conn(0x7f8f34005c20 msgr2=0x7f8f340043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 shutdown_connections 2024-03-29T11:35:52.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 1 -- 172.21.15.145:0/3550939389 wait complete. 2024-03-29T11:35:52.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:52.471+0000 7f8f4209d640 1 librados: shutdown 2024-03-29T11:35:52.491 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:35:59.297 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:35:59.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.445+0000 7fab85c01640 10 monclient: get_monmap_and_config 2024-03-29T11:35:59.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.445+0000 7fab85c01640 10 monclient: build_initial_monmap 2024-03-29T11:35:59.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.445+0000 7fab85c01640 10 monclient: monmap: 2024-03-29T11:35:59.443 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:59.443 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:59.443 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:35:59.445380+0000 2024-03-29T11:35:59.443 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:35:59.445380+0000 2024-03-29T11:35:59.444 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:59.444 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:59.444 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:59.444 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:35:59.444 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:35:59.444 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:59.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.448+0000 7fab85c01640 1 Processor -- start 2024-03-29T11:35:59.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.448+0000 7fab85c01640 1 -- start start 2024-03-29T11:35:59.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.448+0000 7fab85c01640 10 monclient: init 2024-03-29T11:35:59.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:59.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:35:59.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 10 monclient(hunting): picked mon.noname-a con 0x7fab78006e30 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:59.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 10 monclient(hunting): picked mon.noname-c con 0x7fab78035e00 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:59.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 10 monclient(hunting): picked mon.noname-b con 0x7fab780c9670 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:59.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab78035970 con 0x7fab78006e30 2024-03-29T11:35:59.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab78036280 con 0x7fab78035e00 2024-03-29T11:35:59.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab78035640 con 0x7fab780c9670 2024-03-29T11:35:59.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:59.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.449+0000 7fab85c01640 10 monclient(hunting): authenticate will time out at 948.756160s 2024-03-29T11:35:59.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.452+0000 7fab84bff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fab78006e30 0x7fab780cbcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:37337/0 (socket says 172.21.15.145:37337) 2024-03-29T11:35:59.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.452+0000 7fab84bff640 1 -- 172.21.15.145:0/3819622364 learned_addr learned my addr 172.21.15.145:0/3819622364 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:59.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.453+0000 7fab7f7fe640 1 -- 172.21.15.145:0/3819622364 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179704519 0 0) 0x7fab78035970 con 0x7fab78006e30 2024-03-29T11:35:59.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.453+0000 7fab7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:35:59.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.453+0000 7fab7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:59.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.453+0000 7fab7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:59.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.453+0000 7fab7f7fe640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:35:59.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.453+0000 7fab7f7fe640 1 -- 172.21.15.145:0/3819622364 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab70003520 con 0x7fab78006e30 2024-03-29T11:35:59.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.453+0000 7fab7f7fe640 1 -- 172.21.15.145:0/3819622364 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 936829764 0 0) 0x7fab70003520 con 0x7fab78006e30 2024-03-29T11:35:59.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.453+0000 7fab7f7fe640 1 -- 172.21.15.145:0/3819622364 >> v1:172.21.15.145:6790/0 conn(0x7fab780c9670 legacy=0x7fab780d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:35:59.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.453+0000 7fab7f7fe640 1 -- 172.21.15.145:0/3819622364 >> v1:172.21.15.203:6789/0 conn(0x7fab78035e00 legacy=0x7fab780c8f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:35:59.455 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:35:59.455 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 1 -- 172.21.15.145:0/3819622364 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab78034e20 con 0x7fab78006e30 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 1 -- 172.21.15.145:0/3819622364 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fab6c001670 con 0x7fab78006e30 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 10 monclient: dump: 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.456 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab85c01640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab85c01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 1 -- 172.21.15.145:0/3819622364 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab6c0018d0 con 0x7fab78006e30 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab85c01640 10 monclient: get_monmap_and_config success 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.455+0000 7fab85c01640 4 set_mon_vals no callback set 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.456+0000 7fab85c01640 10 monclient: shutdown 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.456+0000 7fab85c01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:59.457 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.456+0000 7fab85c01640 1 -- 172.21.15.145:0/3819622364 >> v1:172.21.15.145:6789/0 conn(0x7fab78006e30 legacy=0x7fab780cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.460+0000 7fab85c01640 1 -- 172.21.15.145:0/3819622364 shutdown_connections 2024-03-29T11:35:59.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.460+0000 7fab85c01640 1 -- 172.21.15.145:0/3819622364 >> 172.21.15.145:0/3819622364 conn(0x7fab78005c00 msgr2=0x7fab78006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:59.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.460+0000 7fab85c01640 1 -- 172.21.15.145:0/3819622364 shutdown_connections 2024-03-29T11:35:59.464 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.461+0000 7fab85c01640 1 -- 172.21.15.145:0/3819622364 wait complete. 2024-03-29T11:35:59.464 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.462+0000 7fab85c01640 10 monclient: build_initial_monmap 2024-03-29T11:35:59.464 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.462+0000 7fab85c01640 10 monclient: monmap: 2024-03-29T11:35:59.464 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:35:59.464 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:35:59.464 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.462+0000 7fab85c01640 1 librados: starting msgr at 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.462+0000 7fab85c01640 1 librados: starting objecter 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.462+0000 7fab85c01640 1 Processor -- start 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.462+0000 7fab85c01640 1 -- start start 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.462+0000 7fab85c01640 1 librados: setting wanted keys 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.462+0000 7fab85c01640 1 librados: calling monclient init 2024-03-29T11:35:59.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.462+0000 7fab85c01640 10 monclient: init 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 10 monclient: _reopen_session rank -1 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 10 monclient(hunting): picked mon.noname-b con 0x7fab78006e30 addr v1:172.21.15.203:6789/0 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 10 monclient(hunting): picked mon.noname-a con 0x7fab78035e00 addr v1:172.21.15.145:6789/0 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 10 monclient(hunting): picked mon.noname-c con 0x7fab780c9670 addr v1:172.21.15.145:6790/0 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab780cf1a0 con 0x7fab78035e00 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab780cdc90 con 0x7fab78006e30 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab780cdf70 con 0x7fab780c9670 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 10 monclient(hunting): _renew_subs 2024-03-29T11:35:59.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.463+0000 7fab85c01640 10 monclient(hunting): authenticate will time out at 948.770748s 2024-03-29T11:35:59.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.469+0000 7fab84bff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fab78006e30 0x7fab780330f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:48339/0 (socket says 172.21.15.145:48339) 2024-03-29T11:35:59.470 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.469+0000 7fab84bff640 1 -- 172.21.15.145:0/3702986017 learned_addr learned my addr 172.21.15.145:0/3702986017 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:35:59.470 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 client.?.objecter ms_handle_connect 0x7fab78006e30 2024-03-29T11:35:59.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 client.?.objecter ms_handle_connect 0x7fab780c9670 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 client.?.objecter ms_handle_connect 0x7fab78035e00 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992658821 0 0) 0x7fab780cdf70 con 0x7fab780c9670 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab64004eb0 con 0x7fab780c9670 2024-03-29T11:35:59.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892652847 0 0) 0x7fab780cf1a0 con 0x7fab78035e00 2024-03-29T11:35:59.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:59.474 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:59.474 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:59.474 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:35:59.474 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab780cdf70 con 0x7fab78035e00 2024-03-29T11:35:59.474 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898310820 0 0) 0x7fab780cdc90 con 0x7fab78006e30 2024-03-29T11:35:59.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:35:59.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:35:59.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.472+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab780cf1a0 con 0x7fab78006e30 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2230520831 0 0) 0x7fab780cdf70 con 0x7fab78035e00 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab780cdc90 con 0x7fab78035e00 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3364149525 0 0) 0x7fab64004eb0 con 0x7fab780c9670 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab64006a40 con 0x7fab780c9670 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fab8004fdd0 con 0x7fab780c9670 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.479 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fab74001720 con 0x7fab78035e00 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:35:59.480 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:59.481 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:59.481 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.481 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.481 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.481 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:59.481 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:59.481 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:59.481 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:59.481 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3679971364 0 0) 0x7fab780cf1a0 con 0x7fab78006e30 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab64008370 con 0x7fab78006e30 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fab6c001630 con 0x7fab78006e30 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.482 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4240373618 0 0) 0x7fab780cdc90 con 0x7fab78035e00 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 >> v1:172.21.15.145:6790/0 conn(0x7fab780c9670 legacy=0x7fab7818dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 >> v1:172.21.15.203:6789/0 conn(0x7fab78006e30 legacy=0x7fab780330f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 monclient: found mon.a 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7802fd60 con 0x7fab78035e00 2024-03-29T11:35:59.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab85c01640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab85c01640 10 monclient: _renew_subs 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab85c01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.473+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7802f0d0 con 0x7fab78035e00 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.474+0000 7fab85c01640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.474+0000 7fab85c01640 10 monclient: _renew_subs 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.474+0000 7fab85c01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.474+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab780cfb30 con 0x7fab78035e00 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.474+0000 7fab85c01640 1 librados: init done 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.475+0000 7fab85c01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.475+0000 7fab85c01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:59.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.475+0000 7fab85c01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.475+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab78007b70 con 0x7fab78035e00 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.479+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab74001c30 con 0x7fab78035e00 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.479+0000 7fab7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.479+0000 7fab7effd640 4 set_mon_vals no callback set 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.479+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fab74001720 con 0x7fab78035e00 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.479+0000 7fab7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.479+0000 7fab7d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.479+0000 7fab7d7fa640 10 monclient: dump: 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:35:59.485 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:35:59.486 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:35:59.486 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:35:59.486 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:35:59.486 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:35:59.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.479+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fab74019500 con 0x7fab78035e00 2024-03-29T11:35:59.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.480+0000 7fab84bff640 10 monclient: get_auth_request con 0x7fab6407f7a0 auth_method 0 2024-03-29T11:35:59.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.480+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fab74083490 con 0x7fab78035e00 2024-03-29T11:35:59.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.480+0000 7fab7d7fa640 10 client.4272.objecter ms_dispatch 0x7fab80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:35:59.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.480+0000 7fab7d7fa640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:35:59.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.480+0000 7fab7d7fa640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:35:59.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.480+0000 7fab7d7fa640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T11:35:59.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.480+0000 7fab7d7fa640 10 client.4272.objecter ms_handle_connect 0x7fab6407f7a0 2024-03-29T11:35:59.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.487+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fab7404edf0 con 0x7fab78035e00 2024-03-29T11:35:59.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.487+0000 7fab7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:35:59.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.487+0000 7fab7d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:35:59.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.807+0000 7fab85c01640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:59.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.807+0000 7fab85c01640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:59.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.807+0000 7fab85c01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:35:59.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.807+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab78005120 con 0x7fab78035e00 2024-03-29T11:35:59.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.810+0000 7fab7d7fa640 1 -- 172.21.15.145:0/3702986017 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fab74050d20 con 0x7fab78035e00 2024-03-29T11:35:59.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.810+0000 7fab7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:35:59.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.810+0000 7fab7d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:35:59.809 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:35:59.809 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:35:59.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.816+0000 7fab85c01640 10 librados: watch_flush enter 2024-03-29T11:35:59.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.816+0000 7fab85c01640 10 librados: watch_flush exit 2024-03-29T11:35:59.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.816+0000 7fab85c01640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T11:35:59.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.816+0000 7fab85c01640 10 client.4272.objecter successfully canceled tick 2024-03-29T11:35:59.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.818+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 >> v1:172.21.15.145:6816/227372239 conn(0x7fab6407f7a0 legacy=0x7fab64081c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.818+0000 7fab85c01640 10 monclient: shutdown 2024-03-29T11:35:59.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.818+0000 7fab85c01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:35:59.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.818+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 >> v1:172.21.15.145:6789/0 conn(0x7fab78035e00 legacy=0x7fab7802f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:35:59.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.820+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 shutdown_connections 2024-03-29T11:35:59.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.820+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 >> 172.21.15.145:0/3702986017 conn(0x7fab78005c00 msgr2=0x7fab780c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:35:59.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.820+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 shutdown_connections 2024-03-29T11:35:59.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.820+0000 7fab85c01640 1 -- 172.21.15.145:0/3702986017 wait complete. 2024-03-29T11:35:59.819 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:35:59.821+0000 7fab85c01640 1 librados: shutdown 2024-03-29T11:35:59.840 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:06.648 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:06.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.797+0000 7f121b415640 10 monclient: get_monmap_and_config 2024-03-29T11:36:06.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.797+0000 7f121b415640 10 monclient: build_initial_monmap 2024-03-29T11:36:06.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.797+0000 7f121b415640 10 monclient: monmap: 2024-03-29T11:36:06.796 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:06.796 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:06.796 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:36:06.798054+0000 2024-03-29T11:36:06.796 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:36:06.798054+0000 2024-03-29T11:36:06.796 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:06.796 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:06.796 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:06.797 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:36:06.797 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:36:06.797 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.799+0000 7f121b415640 1 Processor -- start 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.799+0000 7f121b415640 1 -- start start 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.799+0000 7f121b415640 10 monclient: init 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 10 monclient(hunting): picked mon.noname-a con 0x7f120c006a80 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 10 monclient(hunting): picked mon.noname-c con 0x7f120c030510 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 10 monclient(hunting): picked mon.noname-b con 0x7f120c0cbd60 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c0cc1f0 con 0x7f120c006a80 2024-03-29T11:36:06.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c035690 con 0x7f120c030510 2024-03-29T11:36:06.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c037240 con 0x7f120c0cbd60 2024-03-29T11:36:06.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:06.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.800+0000 7f121b415640 10 monclient(hunting): authenticate will time out at 956.107796s 2024-03-29T11:36:06.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.803+0000 7f12199b2640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f120c0cbd60 0x7f120c0d7600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:37527/0 (socket says 172.21.15.145:37527) 2024-03-29T11:36:06.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.803+0000 7f12199b2640 1 -- 172.21.15.145:0/4158594246 learned_addr learned my addr 172.21.15.145:0/4158594246 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:06.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.804+0000 7f120bfff640 1 -- 172.21.15.145:0/4158594246 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713018291 0 0) 0x7f120c037240 con 0x7f120c0cbd60 2024-03-29T11:36:06.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.804+0000 7f120bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:06.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.804+0000 7f120bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:06.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.804+0000 7f120bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:06.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.804+0000 7f120bfff640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:36:06.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.804+0000 7f120bfff640 1 -- 172.21.15.145:0/4158594246 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1200003520 con 0x7f120c0cbd60 2024-03-29T11:36:06.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.804+0000 7f120bfff640 1 -- 172.21.15.145:0/4158594246 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 513370894 0 0) 0x7f1200003520 con 0x7f120c0cbd60 2024-03-29T11:36:06.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.804+0000 7f120bfff640 1 -- 172.21.15.145:0/4158594246 >> v1:172.21.15.203:6789/0 conn(0x7f120c030510 legacy=0x7f120c033420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:06.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 1 -- 172.21.15.145:0/4158594246 >> v1:172.21.15.145:6789/0 conn(0x7f120c006a80 legacy=0x7f120c035b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 1 monclient: found mon.noname-b 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 1 -- 172.21.15.145:0/4158594246 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c033b30 con 0x7f120c0cbd60 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 10 monclient: _finish_auth 0 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 1 -- 172.21.15.145:0/4158594246 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1210001670 con 0x7f120c0cbd60 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.806+0000 7f120bfff640 10 monclient: dump: 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:06.806 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f121b415640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f121b415640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f120bfff640 1 -- 172.21.15.145:0/4158594246 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12100018d0 con 0x7f120c0cbd60 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f120bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f120bfff640 1 -- 172.21.15.145:0/4158594246 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1210001d20 con 0x7f120c0cbd60 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f120bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:06.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f120bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f120bfff640 10 monclient: dump: 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:06.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f121b415640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:06.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.807+0000 7f121b415640 4 set_mon_vals no callback set 2024-03-29T11:36:06.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.808+0000 7f121b415640 10 monclient: shutdown 2024-03-29T11:36:06.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.808+0000 7f121b415640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:06.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.808+0000 7f121b415640 1 -- 172.21.15.145:0/4158594246 >> v1:172.21.15.145:6790/0 conn(0x7f120c0cbd60 legacy=0x7f120c0d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.808+0000 7f121b415640 1 -- 172.21.15.145:0/4158594246 shutdown_connections 2024-03-29T11:36:06.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.808+0000 7f121b415640 1 -- 172.21.15.145:0/4158594246 >> 172.21.15.145:0/4158594246 conn(0x7f120c005c40 msgr2=0x7f120c006050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.812+0000 7f121b415640 1 -- 172.21.15.145:0/4158594246 shutdown_connections 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.812+0000 7f121b415640 1 -- 172.21.15.145:0/4158594246 wait complete. 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.813+0000 7f121b415640 10 monclient: build_initial_monmap 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.813+0000 7f121b415640 10 monclient: monmap: 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:06.812 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.813+0000 7f121b415640 1 librados: starting msgr at 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.813+0000 7f121b415640 1 librados: starting objecter 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.813+0000 7f121b415640 1 Processor -- start 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 1 -- start start 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 1 librados: setting wanted keys 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 1 librados: calling monclient init 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 10 monclient: init 2024-03-29T11:36:06.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:06.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:06.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 10 monclient(hunting): picked mon.noname-c con 0x7f120c006a80 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:06.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 10 monclient(hunting): picked mon.noname-b con 0x7f120c030510 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:06.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 10 monclient(hunting): picked mon.noname-a con 0x7f120c0cbd60 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:06.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c0c88c0 con 0x7f120c0cbd60 2024-03-29T11:36:06.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c0301d0 con 0x7f120c030510 2024-03-29T11:36:06.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f120c02e3e0 con 0x7f120c006a80 2024-03-29T11:36:06.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.814+0000 7f121b415640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:06.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.815+0000 7f121b415640 10 monclient(hunting): authenticate will time out at 956.122026s 2024-03-29T11:36:06.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.823+0000 7f12191b1640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f120c006a80 0x7f120c02fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:37603/0 (socket says 172.21.15.145:37603) 2024-03-29T11:36:06.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.823+0000 7f12191b1640 1 -- 172.21.15.145:0/80651144 learned_addr learned my addr 172.21.15.145:0/80651144 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:06.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.823+0000 7f1209ffb640 10 client.?.objecter ms_handle_connect 0x7f120c006a80 2024-03-29T11:36:06.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.823+0000 7f1209ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.823+0000 7f1209ffb640 10 client.?.objecter ms_handle_connect 0x7f120c0cbd60 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.823+0000 7f1209ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813356845 0 0) 0x7f120c02e3e0 con 0x7f120c006a80 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f8004eb0 con 0x7f120c006a80 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328798118 0 0) 0x7f120c0c88c0 con 0x7f120c0cbd60 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:36:06.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f8006230 con 0x7f120c0cbd60 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 client.?.objecter ms_handle_connect 0x7f120c030510 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.824+0000 7f1209ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3630856801 0 0) 0x7f11f8004eb0 con 0x7f120c006a80 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f120c0c88c0 con 0x7f120c006a80 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f121404fdd0 con 0x7f120c006a80 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient(hunting): dump: 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:06.829 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335932013 0 0) 0x7f120c0301d0 con 0x7f120c030510 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11f8006a20 con 0x7f120c030510 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2038397637 0 0) 0x7f120c0c88c0 con 0x7f120c006a80 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 >> v1:172.21.15.203:6789/0 conn(0x7f120c030510 legacy=0x7f120c0c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 >> v1:172.21.15.145:6789/0 conn(0x7f120c0cbd60 legacy=0x7f120c02d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:06.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 monclient: found mon.c 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f120c156730 con 0x7f120c006a80 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f1209ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f121b415640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f121b415640 10 monclient: _renew_subs 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f121b415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120c156a00 con 0x7f120c006a80 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f121b415640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:06.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f121b415640 10 monclient: _renew_subs 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f121b415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f120c0c8550 con 0x7f120c006a80 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.825+0000 7f121b415640 1 librados: init done 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.827+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f121404f400 con 0x7f120c006a80 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.827+0000 7f1209ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.827+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1214075710 con 0x7f120c006a80 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.827+0000 7f1209ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.827+0000 7f121b415640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.827+0000 7f1209ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:06.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.827+0000 7f1209ffb640 10 monclient: dump: 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.827+0000 7f120b7fe640 4 set_mon_vals no callback set 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.828+0000 7f121b415640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:06.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.828+0000 7f121b415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:06.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.828+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120c02f570 con 0x7f120c006a80 2024-03-29T11:36:06.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.828+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f1214067bf0 con 0x7f120c006a80 2024-03-29T11:36:06.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.832+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f12140ff340 con 0x7f120c006a80 2024-03-29T11:36:06.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.832+0000 7f12189b0640 10 monclient: get_auth_request con 0x7f11f806e2a0 auth_method 0 2024-03-29T11:36:06.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.832+0000 7f1209ffb640 10 client.4274.objecter ms_dispatch 0x7f1214000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:06.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.832+0000 7f1209ffb640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:06.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.832+0000 7f1209ffb640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:06.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.832+0000 7f1209ffb640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T11:36:06.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.833+0000 7f1209ffb640 10 client.4274.objecter ms_handle_connect 0x7f11f806e2a0 2024-03-29T11:36:06.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.838+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f120c02f570 con 0x7f120c006a80 2024-03-29T11:36:06.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.838+0000 7f1209ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:06.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:06.838+0000 7f1209ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:07.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.165+0000 7f121b415640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:07.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.165+0000 7f121b415640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:07.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.165+0000 7f121b415640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:07.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.165+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f120c004cf0 con 0x7f120c006a80 2024-03-29T11:36:07.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.166+0000 7f1209ffb640 1 -- 172.21.15.145:0/80651144 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f120c02f570 con 0x7f120c006a80 2024-03-29T11:36:07.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.166+0000 7f1209ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:07.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.166+0000 7f1209ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:07.165 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:36:07.165 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:07.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.171+0000 7f121b415640 10 librados: watch_flush enter 2024-03-29T11:36:07.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.171+0000 7f121b415640 10 librados: watch_flush exit 2024-03-29T11:36:07.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.171+0000 7f121b415640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T11:36:07.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.171+0000 7f121b415640 10 client.4274.objecter successfully canceled tick 2024-03-29T11:36:07.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.172+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 >> v1:172.21.15.145:6816/227372239 conn(0x7f11f806e2a0 legacy=0x7f11f8070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.172+0000 7f121b415640 10 monclient: shutdown 2024-03-29T11:36:07.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.172+0000 7f121b415640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:07.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.172+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 >> v1:172.21.15.145:6790/0 conn(0x7f120c006a80 legacy=0x7f120c02fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:07.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.172+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 shutdown_connections 2024-03-29T11:36:07.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.172+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 >> 172.21.15.145:0/80651144 conn(0x7f120c005c40 msgr2=0x7f120c0ca770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:07.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.173+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 shutdown_connections 2024-03-29T11:36:07.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.176+0000 7f121b415640 1 -- 172.21.15.145:0/80651144 wait complete. 2024-03-29T11:36:07.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:07.176+0000 7f121b415640 1 librados: shutdown 2024-03-29T11:36:07.193 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:14.001 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.155+0000 7fda793c3640 10 monclient: get_monmap_and_config 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.155+0000 7fda793c3640 10 monclient: build_initial_monmap 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.155+0000 7fda793c3640 10 monclient: monmap: 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:36:14.157243+0000 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:36:14.157243+0000 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:14.156 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:36:14.157 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:36:14.157 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:14.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.157+0000 7fda793c3640 1 Processor -- start 2024-03-29T11:36:14.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.158+0000 7fda793c3640 1 -- start start 2024-03-29T11:36:14.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.158+0000 7fda793c3640 10 monclient: init 2024-03-29T11:36:14.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:14.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:14.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 10 monclient(hunting): picked mon.noname-b con 0x7fda6c006410 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:14.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 10 monclient(hunting): picked mon.noname-a con 0x7fda6c02a640 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:14.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 10 monclient(hunting): picked mon.noname-c con 0x7fda6c02b370 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:14.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c02a120 con 0x7fda6c02a640 2024-03-29T11:36:14.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c0c75b0 con 0x7fda6c02b370 2024-03-29T11:36:14.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c033110 con 0x7fda6c006410 2024-03-29T11:36:14.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:14.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.159+0000 7fda793c3640 10 monclient(hunting): authenticate will time out at 963.467584s 2024-03-29T11:36:14.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.162+0000 7fda73fff640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fda6c006410 0x7fda6c0345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:52323/0 (socket says 172.21.15.145:52323) 2024-03-29T11:36:14.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.162+0000 7fda73fff640 1 -- 172.21.15.145:0/1059619314 learned_addr learned my addr 172.21.15.145:0/1059619314 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:14.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.163+0000 7fda72ffd640 1 -- 172.21.15.145:0/1059619314 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2664365028 0 0) 0x7fda6c033110 con 0x7fda6c006410 2024-03-29T11:36:14.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.163+0000 7fda72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:36:14.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.163+0000 7fda72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:14.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.163+0000 7fda72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:14.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.163+0000 7fda72ffd640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:36:14.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.163+0000 7fda72ffd640 1 -- 172.21.15.145:0/1059619314 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda64003520 con 0x7fda6c006410 2024-03-29T11:36:14.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.163+0000 7fda72ffd640 1 -- 172.21.15.145:0/1059619314 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1893641440 0 0) 0x7fda64003520 con 0x7fda6c006410 2024-03-29T11:36:14.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.163+0000 7fda72ffd640 1 -- 172.21.15.145:0/1059619314 >> v1:172.21.15.203:6789/0 conn(0x7fda6c02b370 legacy=0x7fda6c0320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:14.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.164+0000 7fda72ffd640 1 -- 172.21.15.145:0/1059619314 >> v1:172.21.15.145:6789/0 conn(0x7fda6c02a640 legacy=0x7fda6c0c6ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 1 monclient: found mon.noname-b 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 1 -- 172.21.15.145:0/1059619314 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0332a0 con 0x7fda6c006410 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: _finish_auth 0 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda793c3640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda793c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 1 -- 172.21.15.145:0/1059619314 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fda68001690 con 0x7fda6c006410 2024-03-29T11:36:14.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: dump: 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda793c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:14.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 1 -- 172.21.15.145:0/1059619314 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda680018b0 con 0x7fda6c006410 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda793c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda793c3640 4 set_mon_vals no callback set 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 1 -- 172.21.15.145:0/1059619314 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fda68001c40 con 0x7fda6c006410 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.168+0000 7fda72ffd640 10 monclient: dump: 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:14.172 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.169+0000 7fda793c3640 10 monclient: shutdown 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.169+0000 7fda793c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.169+0000 7fda793c3640 1 -- 172.21.15.145:0/1059619314 >> v1:172.21.15.145:6790/0 conn(0x7fda6c006410 legacy=0x7fda6c0345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.169+0000 7fda793c3640 1 -- 172.21.15.145:0/1059619314 shutdown_connections 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.169+0000 7fda793c3640 1 -- 172.21.15.145:0/1059619314 >> 172.21.15.145:0/1059619314 conn(0x7fda6c0058f0 msgr2=0x7fda6c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.169+0000 7fda793c3640 1 -- 172.21.15.145:0/1059619314 shutdown_connections 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.169+0000 7fda793c3640 1 -- 172.21.15.145:0/1059619314 wait complete. 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.170+0000 7fda793c3640 10 monclient: build_initial_monmap 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.170+0000 7fda793c3640 10 monclient: monmap: 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:36:14.173 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.170+0000 7fda793c3640 1 librados: starting msgr at 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.170+0000 7fda793c3640 1 librados: starting objecter 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.170+0000 7fda793c3640 1 Processor -- start 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.172+0000 7fda793c3640 1 -- start start 2024-03-29T11:36:14.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.172+0000 7fda793c3640 1 librados: setting wanted keys 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.172+0000 7fda793c3640 1 librados: calling monclient init 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.172+0000 7fda793c3640 10 monclient: init 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 10 monclient(hunting): picked mon.noname-a con 0x7fda6c006410 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 10 monclient(hunting): picked mon.noname-c con 0x7fda6c02a640 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 10 monclient(hunting): picked mon.noname-b con 0x7fda6c02b370 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c0c07c0 con 0x7fda6c006410 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c0d49d0 con 0x7fda6c02b370 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda6c0d3570 con 0x7fda6c02a640 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:14.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.173+0000 7fda793c3640 10 monclient(hunting): authenticate will time out at 963.481184s 2024-03-29T11:36:14.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.177+0000 7fda73fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fda6c006410 0x7fda6c0c00b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:53515/0 (socket says 172.21.15.145:53515) 2024-03-29T11:36:14.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.177+0000 7fda73fff640 1 -- 172.21.15.145:0/866391028 learned_addr learned my addr 172.21.15.145:0/866391028 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.179+0000 7fda70ff9640 10 client.?.objecter ms_handle_connect 0x7fda6c006410 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.179+0000 7fda70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.179+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707190556 0 0) 0x7fda6c0c07c0 con 0x7fda6c006410 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda58004eb0 con 0x7fda6c006410 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 client.?.objecter ms_handle_connect 0x7fda6c02b370 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716110788 0 0) 0x7fda6c0d49d0 con 0x7fda6c02b370 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:36:14.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda6c0c07c0 con 0x7fda6c02b370 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 779429498 0 0) 0x7fda58004eb0 con 0x7fda6c006410 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.180+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda6c0d49d0 con 0x7fda6c006410 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fda680015d0 con 0x7fda6c006410 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 10 monclient(hunting): dump: 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:14.185 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2381176925 0 0) 0x7fda6c0c07c0 con 0x7fda6c02b370 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda58004eb0 con 0x7fda6c02b370 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2177445805 0 0) 0x7fda6c0d49d0 con 0x7fda6c006410 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 >> v1:172.21.15.145:6790/0 conn(0x7fda6c02a640 legacy=0x7fda6c0d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 >> v1:172.21.15.203:6789/0 conn(0x7fda6c02b370 legacy=0x7fda6c18d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 1 monclient: found mon.a 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:14.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.181+0000 7fda70ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda6c0bd810 con 0x7fda6c006410 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda70ff9640 10 monclient: _finish_auth 0 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda70ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda793c3640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda793c3640 10 monclient: _renew_subs 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda793c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda6c0d3850 con 0x7fda6c006410 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda793c3640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda793c3640 10 monclient: _renew_subs 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda793c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fda6c0d4fd0 con 0x7fda6c006410 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda793c3640 1 librados: init done 2024-03-29T11:36:14.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fda68001730 con 0x7fda6c006410 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.182+0000 7fda71ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.183+0000 7fda793c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.183+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fda68004c30 con 0x7fda6c006410 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.183+0000 7fda70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.183+0000 7fda70ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.183+0000 7fda70ff9640 10 monclient: dump: 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:14.188 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:14.189 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:14.189 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:14.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.183+0000 7fda793c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:14.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.184+0000 7fda793c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:36:14.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.184+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda54005080 con 0x7fda6c006410 2024-03-29T11:36:14.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.185+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fda6801ac00 con 0x7fda6c006410 2024-03-29T11:36:14.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.186+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fda68084bd0 con 0x7fda6c006410 2024-03-29T11:36:14.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.186+0000 7fda70ff9640 10 client.4281.objecter ms_dispatch 0x7fda74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:14.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.186+0000 7fda70ff9640 3 client.4281.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:14.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.186+0000 7fda70ff9640 3 client.4281.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:14.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.186+0000 7fda70ff9640 20 client.4281.objecter dump_active .. 0 homeless 2024-03-29T11:36:14.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.186+0000 7fda737fe640 10 monclient: get_auth_request con 0x7fda5806dc70 auth_method 0 2024-03-29T11:36:14.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.187+0000 7fda70ff9640 10 client.4281.objecter ms_handle_connect 0x7fda5806dc70 2024-03-29T11:36:14.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.193+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fda680504b0 con 0x7fda6c006410 2024-03-29T11:36:14.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.193+0000 7fda70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:14.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.193+0000 7fda70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:14.512 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.513+0000 7fda793c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:14.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.513+0000 7fda793c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:14.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.513+0000 7fda793c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:36:14.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.513+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda54005330 con 0x7fda6c006410 2024-03-29T11:36:14.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.520+0000 7fda70ff9640 1 -- 172.21.15.145:0/866391028 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fda680523e0 con 0x7fda6c006410 2024-03-29T11:36:14.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.520+0000 7fda70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:14.520 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.520+0000 7fda70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:14.522 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:36:14.522 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:14.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.527+0000 7fda793c3640 10 librados: watch_flush enter 2024-03-29T11:36:14.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.529+0000 7fda793c3640 10 librados: watch_flush exit 2024-03-29T11:36:14.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.529+0000 7fda793c3640 20 client.4281.objecter shutdown clearing up homeless session... 2024-03-29T11:36:14.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.529+0000 7fda793c3640 10 client.4281.objecter successfully canceled tick 2024-03-29T11:36:14.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.529+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 >> v1:172.21.15.145:6816/227372239 conn(0x7fda5806dc70 legacy=0x7fda58070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.529+0000 7fda793c3640 10 monclient: shutdown 2024-03-29T11:36:14.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.529+0000 7fda793c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:14.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.529+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 >> v1:172.21.15.145:6789/0 conn(0x7fda6c006410 legacy=0x7fda6c0c00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:14.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.529+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 shutdown_connections 2024-03-29T11:36:14.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.529+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 >> 172.21.15.145:0/866391028 conn(0x7fda6c0058f0 msgr2=0x7fda6c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:14.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.531+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 shutdown_connections 2024-03-29T11:36:14.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.535+0000 7fda793c3640 1 -- 172.21.15.145:0/866391028 wait complete. 2024-03-29T11:36:14.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:14.535+0000 7fda793c3640 1 librados: shutdown 2024-03-29T11:36:14.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:21.358 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:21.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.517+0000 7f107dc57640 10 monclient: get_monmap_and_config 2024-03-29T11:36:21.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.517+0000 7f107dc57640 10 monclient: build_initial_monmap 2024-03-29T11:36:21.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.517+0000 7f107dc57640 10 monclient: monmap: 2024-03-29T11:36:21.517 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:21.517 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:21.517 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:36:21.519097+0000 2024-03-29T11:36:21.517 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:36:21.519097+0000 2024-03-29T11:36:21.517 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:21.518 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:21.518 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:21.518 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:36:21.518 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:36:21.518 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:21.518 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.519+0000 7f107dc57640 1 Processor -- start 2024-03-29T11:36:21.518 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.519+0000 7f107dc57640 1 -- start start 2024-03-29T11:36:21.518 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.519+0000 7f107dc57640 10 monclient: init 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 10 monclient(hunting): picked mon.noname-a con 0x7f10780b4330 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 10 monclient(hunting): picked mon.noname-c con 0x7f10781470b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 10 monclient(hunting): picked mon.noname-b con 0x7f10781439b0 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107806f2a0 con 0x7f10780b4330 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1078146b90 con 0x7f10781470b0 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10780b2ca0 con 0x7f10781439b0 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:21.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f107dc57640 10 monclient(hunting): authenticate will time out at 970.828211s 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f10777fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f10780b4330 0x7f10780b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:37088/0 (socket says 172.21.15.145:37088) 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f1076ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f10781470b0 0x7f10780b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:40638/0 (socket says 172.21.15.145:40638) 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f10777fe640 1 -- 172.21.15.145:0/2998739304 learned_addr learned my addr 172.21.15.145:0/2998739304 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598300601 0 0) 0x7f107806f2a0 con 0x7f10780b4330 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f10767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f10767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f10767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.520+0000 7f10767fc640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1060003520 con 0x7f10780b4330 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486339333 0 0) 0x7f1078146b90 con 0x7f10781470b0 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f107806f2a0 con 0x7f10781470b0 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009125629 0 0) 0x7f10780b2ca0 con 0x7f10781439b0 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1078146b90 con 0x7f10781439b0 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3019262278 0 0) 0x7f107806f2a0 con 0x7f10781470b0 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 >> v1:172.21.15.145:6790/0 conn(0x7f10781439b0 legacy=0x7f1078151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 >> v1:172.21.15.145:6789/0 conn(0x7f10780b4330 legacy=0x7f10780b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 monclient: found mon.noname-c 2024-03-29T11:36:21.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:21.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:36:21.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107806e0c0 con 0x7f10781470b0 2024-03-29T11:36:21.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:21.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:21.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:21.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f106c001670 con 0x7f10781470b0 2024-03-29T11:36:21.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:21.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:21.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient: dump: 2024-03-29T11:36:21.524 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:21.524 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:21.524 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:21.524 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:21.524 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:21.524 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f107dc57640 5 monclient: authenticate success, global_id 4270 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f107dc57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f106c001890 con 0x7f10781470b0 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.521+0000 7f10767fc640 1 -- 172.21.15.145:0/2998739304 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f106c001c20 con 0x7f10781470b0 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.522+0000 7f10767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.522+0000 7f10767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.522+0000 7f10767fc640 10 monclient: dump: 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:21.525 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.522+0000 7f107dc57640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:21.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.522+0000 7f107dc57640 4 set_mon_vals no callback set 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.523+0000 7f107dc57640 10 monclient: shutdown 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.523+0000 7f107dc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.523+0000 7f107dc57640 1 -- 172.21.15.145:0/2998739304 >> v1:172.21.15.203:6789/0 conn(0x7f10781470b0 legacy=0x7f10780b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.523+0000 7f107dc57640 1 -- 172.21.15.145:0/2998739304 shutdown_connections 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.523+0000 7f107dc57640 1 -- 172.21.15.145:0/2998739304 >> 172.21.15.145:0/2998739304 conn(0x7f1078066a00 msgr2=0x7f1078066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.523+0000 7f107dc57640 1 -- 172.21.15.145:0/2998739304 shutdown_connections 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.523+0000 7f107dc57640 1 -- 172.21.15.145:0/2998739304 wait complete. 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.524+0000 7f107dc57640 10 monclient: build_initial_monmap 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.524+0000 7f107dc57640 10 monclient: monmap: 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:21.530 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.524+0000 7f107dc57640 1 librados: starting msgr at 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.524+0000 7f107dc57640 1 librados: starting objecter 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.524+0000 7f107dc57640 1 Processor -- start 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.525+0000 7f107dc57640 1 -- start start 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.525+0000 7f107dc57640 1 librados: setting wanted keys 2024-03-29T11:36:21.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.525+0000 7f107dc57640 1 librados: calling monclient init 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.525+0000 7f107dc57640 10 monclient: init 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f107dc57640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f107dc57640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f107dc57640 10 monclient(hunting): picked mon.noname-c con 0x7f10780b4330 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f107dc57640 10 monclient(hunting): picked mon.noname-b con 0x7f10781439b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f107dc57640 10 monclient(hunting): picked mon.noname-a con 0x7f10781470b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f107dc57640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1078127520 con 0x7f10781470b0 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f1076ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f10781439b0 0x7f1078141270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:40886/0 (socket says 172.21.15.145:40886) 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f10777fe640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f10780b4330 0x7f10781dbbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:43032/0 (socket says 172.21.15.145:43032) 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f1076ffd640 1 -- 172.21.15.145:0/4151058229 learned_addr learned my addr 172.21.15.145:0/4151058229 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1078122740 con 0x7f10781439b0 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10781228d0 con 0x7f10780b4330 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.526+0000 7f107dc57640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f107dc57640 10 monclient(hunting): authenticate will time out at 970.835031s 2024-03-29T11:36:21.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 client.?.objecter ms_handle_connect 0x7f10781439b0 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762489831 0 0) 0x7f1078122740 con 0x7f10781439b0 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1064004eb0 con 0x7f10781439b0 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 client.?.objecter ms_handle_connect 0x7f10781470b0 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 client.?.objecter ms_handle_connect 0x7f10780b4330 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117298595 0 0) 0x7f1078127520 con 0x7f10781470b0 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1078122740 con 0x7f10781470b0 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 993781762 0 0) 0x7f1064004eb0 con 0x7f10781439b0 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.527+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1078127520 con 0x7f10781439b0 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f106c001670 con 0x7f10781439b0 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient(hunting): dump: 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:21.534 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859081354 0 0) 0x7f10781228d0 con 0x7f10780b4330 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1064004eb0 con 0x7f10780b4330 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 901759419 0 0) 0x7f1078127520 con 0x7f10781439b0 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 >> v1:172.21.15.145:6790/0 conn(0x7f10780b4330 legacy=0x7f10781dbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 >> v1:172.21.15.145:6789/0 conn(0x7f10781470b0 legacy=0x7f1078125af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 1 monclient: found mon.b 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10781dc2d0 con 0x7f10781439b0 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient: _finish_auth 0 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f105ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f107dc57640 5 monclient: authenticate success, global_id 4273 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f107dc57640 10 monclient: _renew_subs 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f107dc57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10781dc460 con 0x7f10781439b0 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f107dc57640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f107dc57640 10 monclient: _renew_subs 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f107dc57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:21.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1078141b60 con 0x7f10781439b0 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.528+0000 7f107dc57640 1 librados: init done 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.529+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f106c004a90 con 0x7f10781439b0 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.529+0000 7f105ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.529+0000 7f1075ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.529+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f106c004e20 con 0x7f10781439b0 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.529+0000 7f105ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.529+0000 7f105ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.529+0000 7f105ffff640 10 monclient: dump: 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:21.537 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:21.538 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:21.538 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:21.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.530+0000 7f107dc57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:21.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.530+0000 7f107dc57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:21.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.530+0000 7f107dc57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:21.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.530+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1058005080 con 0x7f10781439b0 2024-03-29T11:36:21.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.533+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f106c004f80 con 0x7f10781439b0 2024-03-29T11:36:21.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.534+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f106c084ba0 con 0x7f10781439b0 2024-03-29T11:36:21.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.534+0000 7f105ffff640 10 client.4273.objecter ms_dispatch 0x7f1078000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:21.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.534+0000 7f105ffff640 3 client.4273.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:21.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.534+0000 7f105ffff640 3 client.4273.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:21.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.534+0000 7f105ffff640 20 client.4273.objecter dump_active .. 0 homeless 2024-03-29T11:36:21.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.534+0000 7f10777fe640 10 monclient: get_auth_request con 0x7f106406de00 auth_method 0 2024-03-29T11:36:21.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.535+0000 7f105ffff640 10 client.4273.objecter ms_handle_connect 0x7f106406de00 2024-03-29T11:36:21.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.539+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f106c050480 con 0x7f10781439b0 2024-03-29T11:36:21.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.539+0000 7f105ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:21.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.539+0000 7f105ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:21.925 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.926+0000 7f107dc57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:21.925 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.926+0000 7f107dc57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:21.925 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.926+0000 7f107dc57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:21.925 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.926+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1058005330 con 0x7f10781439b0 2024-03-29T11:36:21.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.927+0000 7f105ffff640 1 -- 172.21.15.145:0/4151058229 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f106c0523b0 con 0x7f10781439b0 2024-03-29T11:36:21.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.927+0000 7f105ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:21.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.927+0000 7f105ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:21.926 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:36:21.926 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:21.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 10 librados: watch_flush enter 2024-03-29T11:36:21.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 10 librados: watch_flush exit 2024-03-29T11:36:21.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 20 client.4273.objecter shutdown clearing up homeless session... 2024-03-29T11:36:21.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 10 client.4273.objecter successfully canceled tick 2024-03-29T11:36:21.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 >> v1:172.21.15.145:6816/227372239 conn(0x7f106406de00 legacy=0x7f10640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 10 monclient: shutdown 2024-03-29T11:36:21.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:21.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 >> v1:172.21.15.203:6789/0 conn(0x7f10781439b0 legacy=0x7f1078141270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:21.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 shutdown_connections 2024-03-29T11:36:21.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.931+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 >> 172.21.15.145:0/4151058229 conn(0x7f1078066a00 msgr2=0x7f1078145b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:21.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.932+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 shutdown_connections 2024-03-29T11:36:21.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.932+0000 7f107dc57640 1 -- 172.21.15.145:0/4151058229 wait complete. 2024-03-29T11:36:21.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:21.932+0000 7f107dc57640 1 librados: shutdown 2024-03-29T11:36:21.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:28.755 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:28.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.891+0000 7f322abd0640 10 monclient: get_monmap_and_config 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.891+0000 7f322abd0640 10 monclient: build_initial_monmap 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.891+0000 7f322abd0640 10 monclient: monmap: 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:36:28.892823+0000 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:36:28.892823+0000 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:36:28.891 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:28.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.892+0000 7f322abd0640 1 Processor -- start 2024-03-29T11:36:28.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.892+0000 7f322abd0640 1 -- start start 2024-03-29T11:36:28.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.892+0000 7f322abd0640 10 monclient: init 2024-03-29T11:36:28.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:28.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:28.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 10 monclient(hunting): picked mon.noname-c con 0x7f32240b4710 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:28.894 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 10 monclient(hunting): picked mon.noname-a con 0x7f32240b3970 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:28.894 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322896c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f32240b4710 0x7f32240b1010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:42306/0 (socket says 172.21.15.145:42306) 2024-03-29T11:36:28.894 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322896c640 1 -- 172.21.15.145:0/1408681776 learned_addr learned my addr 172.21.15.145:0/1408681776 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:28.894 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 10 monclient(hunting): picked mon.noname-b con 0x7f322413f390 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:28.894 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 1 -- 172.21.15.145:0/1408681776 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32240af020 con 0x7f32240b3970 2024-03-29T11:36:28.894 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 1 -- 172.21.15.145:0/1408681776 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32240b0910 con 0x7f32240b4710 2024-03-29T11:36:28.894 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 1 -- 172.21.15.145:0/1408681776 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32240b0bf0 con 0x7f322413f390 2024-03-29T11:36:28.894 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.893+0000 7f322abd0640 10 monclient(hunting): authenticate will time out at 978.201756s 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199706440 0 0) 0x7f32240b0910 con 0x7f32240b4710 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3214003520 con 0x7f32240b4710 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099345488 0 0) 0x7f32240b0bf0 con 0x7f322413f390 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:28.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:36:28.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32240b0910 con 0x7f322413f390 2024-03-29T11:36:28.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 743513992 0 0) 0x7f32240b0910 con 0x7f322413f390 2024-03-29T11:36:28.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 >> v1:172.21.15.203:6789/0 conn(0x7f32240b4710 legacy=0x7f32240b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:28.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.894+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 >> v1:172.21.15.145:6789/0 conn(0x7f32240b3970 legacy=0x7f32240af4b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:28.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:28.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 1 monclient: found mon.noname-b 2024-03-29T11:36:28.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32240aeb00 con 0x7f322413f390 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f321c0019f0 con 0x7f322413f390 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f32237fe640 10 monclient: dump: 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:28.898 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f322abd0640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.896+0000 7f322abd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.897+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f321c001c40 con 0x7f322413f390 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.897+0000 7f32237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.897+0000 7f32237fe640 1 -- 172.21.15.145:0/1408681776 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f321c001fd0 con 0x7f322413f390 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.897+0000 7f322abd0640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:28.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.897+0000 7f322abd0640 4 set_mon_vals no callback set 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.897+0000 7f32237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.897+0000 7f32237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.897+0000 7f32237fe640 10 monclient: dump: 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:28.900 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:28.901 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:28.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.898+0000 7f322abd0640 10 monclient: shutdown 2024-03-29T11:36:28.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.898+0000 7f322abd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:28.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.898+0000 7f322abd0640 1 -- 172.21.15.145:0/1408681776 >> v1:172.21.15.145:6790/0 conn(0x7f322413f390 legacy=0x7f322413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:28.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.898+0000 7f322abd0640 1 -- 172.21.15.145:0/1408681776 shutdown_connections 2024-03-29T11:36:28.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.898+0000 7f322abd0640 1 -- 172.21.15.145:0/1408681776 >> 172.21.15.145:0/1408681776 conn(0x7f3224066a00 msgr2=0x7f3224066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:28.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.898+0000 7f322abd0640 1 -- 172.21.15.145:0/1408681776 shutdown_connections 2024-03-29T11:36:28.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.898+0000 7f322abd0640 1 -- 172.21.15.145:0/1408681776 wait complete. 2024-03-29T11:36:28.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.899+0000 7f322abd0640 10 monclient: build_initial_monmap 2024-03-29T11:36:28.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.899+0000 7f322abd0640 10 monclient: monmap: 2024-03-29T11:36:28.907 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:28.907 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:28.907 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:36:28.907 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:36:28.907 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.899+0000 7f322abd0640 1 librados: starting msgr at 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.899+0000 7f322abd0640 1 librados: starting objecter 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.899+0000 7f322abd0640 1 Processor -- start 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.899+0000 7f322abd0640 1 -- start start 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.899+0000 7f322abd0640 1 librados: setting wanted keys 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.899+0000 7f322abd0640 1 librados: calling monclient init 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.899+0000 7f322abd0640 10 monclient: init 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.900+0000 7f322abd0640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.900+0000 7f322abd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:28.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.900+0000 7f322abd0640 10 monclient(hunting): picked mon.noname-b con 0x7f32240b3970 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f322abd0640 10 monclient(hunting): picked mon.noname-a con 0x7f32240b4710 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f322abd0640 10 monclient(hunting): picked mon.noname-c con 0x7f322413f390 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f322abd0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32241390c0 con 0x7f32240b4710 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f322abd0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3224137520 con 0x7f32240b3970 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f322abd0640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32241377c0 con 0x7f322413f390 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f322abd0640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f322abd0640 10 monclient(hunting): authenticate will time out at 978.209658s 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f322916d640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f322413f390 0x7f32241e28e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:60774/0 (socket says 172.21.15.145:60774) 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f322916d640 1 -- 172.21.15.145:0/2608955378 learned_addr learned my addr 172.21.15.145:0/2608955378 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f32217fa640 10 client.?.objecter ms_handle_connect 0x7f322413f390 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.901+0000 7f32217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356216467 0 0) 0x7f32241377c0 con 0x7f322413f390 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:28.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3208004eb0 con 0x7f322413f390 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2068700649 0 0) 0x7f3208004eb0 con 0x7f322413f390 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32241377c0 con 0x7f322413f390 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f321c001a70 con 0x7f322413f390 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 monclient(hunting): dump: 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:28.910 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 client.?.objecter ms_handle_connect 0x7f32240b3970 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3111534483 0 0) 0x7f32241377c0 con 0x7f322413f390 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 >> v1:172.21.15.203:6789/0 conn(0x7f32240b3970 legacy=0x7f3224133d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.902+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 >> v1:172.21.15.145:6789/0 conn(0x7f32240b4710 legacy=0x7f32241da580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f32217fa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f32217fa640 1 monclient: found mon.c 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f32217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f32217fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:28.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224137ce0 con 0x7f322413f390 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f32217fa640 10 monclient: _finish_auth 0 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f32217fa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f32217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f322abd0640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f322abd0640 10 monclient: _renew_subs 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f322abd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32241344a0 con 0x7f322413f390 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f322abd0640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f322abd0640 10 monclient: _renew_subs 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f322abd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3224139650 con 0x7f322413f390 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.903+0000 7f322abd0640 1 librados: init done 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.904+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f321c003c90 con 0x7f322413f390 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.904+0000 7f32217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:28.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.904+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f321c0040a0 con 0x7f322413f390 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.904+0000 7f32217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.904+0000 7f32217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.904+0000 7f32217fa640 10 monclient: dump: 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.904+0000 7f322abd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:28.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.905+0000 7f322abd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.905+0000 7f322abd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.905+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32240688a0 con 0x7f322413f390 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.905+0000 7f3222ffd640 4 set_mon_vals no callback set 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.905+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f321c004200 con 0x7f322413f390 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.906+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f321c084fe0 con 0x7f322413f390 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.906+0000 7f32217fa640 10 client.4289.objecter ms_dispatch 0x7f3224000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.906+0000 7f32217fa640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.906+0000 7f32217fa640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.906+0000 7f32217fa640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.906+0000 7f322896c640 10 monclient: get_auth_request con 0x7f320806dc40 auth_method 0 2024-03-29T11:36:28.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.906+0000 7f32217fa640 10 client.4289.objecter ms_handle_connect 0x7f320806dc40 2024-03-29T11:36:28.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.913+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f321c050920 con 0x7f322413f390 2024-03-29T11:36:28.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.913+0000 7f32217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:28.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:28.913+0000 7f32217fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:29.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.193+0000 7f322abd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:29.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.193+0000 7f322abd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:29.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.193+0000 7f322abd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:29.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.193+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f32241346d0 con 0x7f322413f390 2024-03-29T11:36:29.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.194+0000 7f32217fa640 1 -- 172.21.15.145:0/2608955378 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f321c084690 con 0x7f322413f390 2024-03-29T11:36:29.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.194+0000 7f32217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:29.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.194+0000 7f32217fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:29.194 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:36:29.194 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:29.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.198+0000 7f322abd0640 10 librados: watch_flush enter 2024-03-29T11:36:29.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.198+0000 7f322abd0640 10 librados: watch_flush exit 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.198+0000 7f322abd0640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.198+0000 7f322abd0640 10 client.4289.objecter successfully canceled tick 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.199+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 >> v1:172.21.15.145:6816/227372239 conn(0x7f320806dc40 legacy=0x7f3208070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.199+0000 7f322abd0640 10 monclient: shutdown 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.199+0000 7f322abd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.199+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 >> v1:172.21.15.145:6790/0 conn(0x7f322413f390 legacy=0x7f32241e28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.199+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 shutdown_connections 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.199+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 >> 172.21.15.145:0/2608955378 conn(0x7f3224066a00 msgr2=0x7f32240b1770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.199+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 shutdown_connections 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.199+0000 7f322abd0640 1 -- 172.21.15.145:0/2608955378 wait complete. 2024-03-29T11:36:29.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:29.199+0000 7f322abd0640 1 librados: shutdown 2024-03-29T11:36:29.211 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:36.019 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.164+0000 7f2623e74640 10 monclient: get_monmap_and_config 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.164+0000 7f2623e74640 10 monclient: build_initial_monmap 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.164+0000 7f2623e74640 10 monclient: monmap: 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:36:36.165703+0000 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:36:36.165703+0000 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:36:36.164 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:36.165 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.165+0000 7f2623e74640 1 Processor -- start 2024-03-29T11:36:36.165 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.166+0000 7f2623e74640 1 -- start start 2024-03-29T11:36:36.165 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.166+0000 7f2623e74640 10 monclient: init 2024-03-29T11:36:36.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:36.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:36.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 10 monclient(hunting): picked mon.noname-c con 0x7f2614006e30 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:36.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 10 monclient(hunting): picked mon.noname-a con 0x7f2614035e00 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:36.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 10 monclient(hunting): picked mon.noname-b con 0x7f26140c9670 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:36.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2614035970 con 0x7f2614035e00 2024-03-29T11:36:36.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2614036280 con 0x7f2614006e30 2024-03-29T11:36:36.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2614035640 con 0x7f26140c9670 2024-03-29T11:36:36.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:36.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.167+0000 7f2623e74640 10 monclient(hunting): authenticate will time out at 985.475609s 2024-03-29T11:36:36.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.170+0000 7f2622411640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f26140c9670 0x7f26140d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:33633/0 (socket says 172.21.15.145:33633) 2024-03-29T11:36:36.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.170+0000 7f2622411640 1 -- 172.21.15.145:0/1954608153 learned_addr learned my addr 172.21.15.145:0/1954608153 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:36.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.170+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499400310 0 0) 0x7f2614035640 con 0x7f26140c9670 2024-03-29T11:36:36.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.170+0000 7f2620c0e640 10 monclient(hunting): _negotiate 2024-03-29T11:36:36.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.170+0000 7f2620c0e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:36.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.170+0000 7f2620c0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:36.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.170+0000 7f2620c0e640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:36:36.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.170+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2608003520 con 0x7f26140c9670 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746827229 0 0) 0x7f2614036280 con 0x7f2614006e30 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient(hunting): _negotiate 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2614035640 con 0x7f2614006e30 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2573722125 0 0) 0x7f2608003520 con 0x7f26140c9670 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 >> v1:172.21.15.203:6789/0 conn(0x7f2614006e30 legacy=0x7f26140cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 >> v1:172.21.15.145:6789/0 conn(0x7f2614035e00 legacy=0x7f26140c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient: _finish_hunting 0 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 1 monclient: found mon.noname-b 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2614034e20 con 0x7f26140c9670 2024-03-29T11:36:36.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient: _finish_auth 0 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient: _check_auth_tickets 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f261c04f730 con 0x7f26140c9670 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.171+0000 7f2620c0e640 10 monclient: dump: 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:36.177 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.172+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f261c075f10 con 0x7f26140c9670 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.172+0000 7f2620c0e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.172+0000 7f2620c0e640 1 -- 172.21.15.145:0/1954608153 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f261c0758b0 con 0x7f26140c9670 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.172+0000 7f2620c0e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.172+0000 7f2620c0e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.172+0000 7f2620c0e640 10 monclient: dump: 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:36.178 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.172+0000 7f2623e74640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.172+0000 7f2623e74640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.172+0000 7f2623e74640 4 set_mon_vals no callback set 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.173+0000 7f2623e74640 10 monclient: shutdown 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.173+0000 7f2623e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.173+0000 7f2623e74640 1 -- 172.21.15.145:0/1954608153 >> v1:172.21.15.145:6790/0 conn(0x7f26140c9670 legacy=0x7f26140d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.173+0000 7f2623e74640 1 -- 172.21.15.145:0/1954608153 shutdown_connections 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.173+0000 7f2623e74640 1 -- 172.21.15.145:0/1954608153 >> 172.21.15.145:0/1954608153 conn(0x7f2614005c00 msgr2=0x7f2614006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.173+0000 7f2623e74640 1 -- 172.21.15.145:0/1954608153 shutdown_connections 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.174+0000 7f2623e74640 1 -- 172.21.15.145:0/1954608153 wait complete. 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.174+0000 7f2623e74640 10 monclient: build_initial_monmap 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.174+0000 7f2623e74640 10 monclient: monmap: 2024-03-29T11:36:36.179 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.175+0000 7f2623e74640 1 librados: starting msgr at 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.175+0000 7f2623e74640 1 librados: starting objecter 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.175+0000 7f2623e74640 1 Processor -- start 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.175+0000 7f2623e74640 1 -- start start 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.175+0000 7f2623e74640 1 librados: setting wanted keys 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.175+0000 7f2623e74640 1 librados: calling monclient init 2024-03-29T11:36:36.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.175+0000 7f2623e74640 10 monclient: init 2024-03-29T11:36:36.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:36.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 10 monclient(hunting): picked mon.noname-b con 0x7f2614006e30 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 10 monclient(hunting): picked mon.noname-a con 0x7f2614035e00 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 10 monclient(hunting): picked mon.noname-c con 0x7f26140c9670 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261402ec30 con 0x7f2614035e00 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261402ef10 con 0x7f2614006e30 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f261402f640 con 0x7f26140c9670 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.176+0000 7f2623e74640 10 monclient(hunting): authenticate will time out at 985.484313s 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.179+0000 7f262140f640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f2614035e00 0x7f261402fb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:40413/0 (socket says 172.21.15.145:40413) 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.179+0000 7f262140f640 1 -- 172.21.15.145:0/1982914561 learned_addr learned my addr 172.21.15.145:0/1982914561 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.179+0000 7f26127fc640 10 client.?.objecter ms_handle_connect 0x7f2614006e30 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.179+0000 7f26127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:36.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044197927 0 0) 0x7f261402ef10 con 0x7f2614006e30 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2600004eb0 con 0x7f2614006e30 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 client.?.objecter ms_handle_connect 0x7f2614035e00 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 client.?.objecter ms_handle_connect 0x7f26140c9670 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155188122 0 0) 0x7f261402f640 con 0x7f26140c9670 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:36:36.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.181+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261402ef10 con 0x7f26140c9670 2024-03-29T11:36:36.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218513132 0 0) 0x7f261402ec30 con 0x7f2614035e00 2024-03-29T11:36:36.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:36.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:36.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:36.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:36:36.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f261402f640 con 0x7f2614035e00 2024-03-29T11:36:36.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747655221 0 0) 0x7f2600004eb0 con 0x7f2614006e30 2024-03-29T11:36:36.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f261402ec30 con 0x7f2614006e30 2024-03-29T11:36:36.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f260c001930 con 0x7f2614006e30 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): dump: 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:36.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 523352199 0 0) 0x7f261402ef10 con 0x7f26140c9670 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2600004eb0 con 0x7f26140c9670 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f261c04f400 con 0x7f26140c9670 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient(hunting): dump: 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:36.191 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3482064563 0 0) 0x7f2600004eb0 con 0x7f26140c9670 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 >> v1:172.21.15.203:6789/0 conn(0x7f2614006e30 legacy=0x7f2614169450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 >> v1:172.21.15.145:6789/0 conn(0x7f2614035e00 legacy=0x7f261402fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 monclient: found mon.c 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261402f330 con 0x7f26140c9670 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2623e74640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2623e74640 10 monclient: _renew_subs 2024-03-29T11:36:36.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2623e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f261402d370 con 0x7f26140c9670 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f261c04f730 con 0x7f26140c9670 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f26127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2623e74640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2613fff640 4 set_mon_vals no callback set 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2623e74640 10 monclient: _renew_subs 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2623e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26140337a0 con 0x7f26140c9670 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.182+0000 7f2623e74640 1 librados: init done 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.183+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f261c067d90 con 0x7f26140c9670 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.183+0000 7f26127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.183+0000 7f26127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.183+0000 7f26127fc640 10 monclient: dump: 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:36.193 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.184+0000 7f2623e74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.184+0000 7f2623e74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.184+0000 7f2623e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.184+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2614007b90 con 0x7f26140c9670 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.184+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f261c067ef0 con 0x7f26140c9670 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.185+0000 7f2621c10640 10 monclient: get_auth_request con 0x7f260006dec0 auth_method 0 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.185+0000 7f26127fc640 10 client.4295.objecter ms_handle_connect 0x7f260006dec0 2024-03-29T11:36:36.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.186+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f261c0ff260 con 0x7f26140c9670 2024-03-29T11:36:36.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.186+0000 7f26127fc640 10 client.4295.objecter ms_dispatch 0x7f261c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:36.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.186+0000 7f26127fc640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:36.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.186+0000 7f26127fc640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:36.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.186+0000 7f26127fc640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-29T11:36:36.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.194+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f261c0cab40 con 0x7f26140c9670 2024-03-29T11:36:36.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.194+0000 7f26127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:36.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.194+0000 7f26127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:36.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.525+0000 7f2623e74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:36.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.525+0000 7f2623e74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:36.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.525+0000 7f2623e74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:36.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.525+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f261402f7d0 con 0x7f26140c9670 2024-03-29T11:36:36.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.526+0000 7f26127fc640 1 -- 172.21.15.145:0/1982914561 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f261c0cca70 con 0x7f26140c9670 2024-03-29T11:36:36.526 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:36:36.526 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:36.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.526+0000 7f26127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:36.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.526+0000 7f26127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:36.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.529+0000 7f2623e74640 10 librados: watch_flush enter 2024-03-29T11:36:36.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.529+0000 7f2623e74640 10 librados: watch_flush exit 2024-03-29T11:36:36.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.529+0000 7f2623e74640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-29T11:36:36.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.529+0000 7f2623e74640 10 client.4295.objecter successfully canceled tick 2024-03-29T11:36:36.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.531+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 >> v1:172.21.15.145:6816/227372239 conn(0x7f260006dec0 legacy=0x7f2600070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.531+0000 7f2623e74640 10 monclient: shutdown 2024-03-29T11:36:36.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.531+0000 7f2623e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:36.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.531+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 >> v1:172.21.15.145:6790/0 conn(0x7f26140c9670 legacy=0x7f2614169d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:36.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.532+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 shutdown_connections 2024-03-29T11:36:36.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.532+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 >> 172.21.15.145:0/1982914561 conn(0x7f2614005c00 msgr2=0x7f2614005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:36.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.532+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 shutdown_connections 2024-03-29T11:36:36.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.532+0000 7f2623e74640 1 -- 172.21.15.145:0/1982914561 wait complete. 2024-03-29T11:36:36.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:36.532+0000 7f2623e74640 1 librados: shutdown 2024-03-29T11:36:36.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:43.357 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:43.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.515+0000 7f0af7fff640 10 monclient: get_monmap_and_config 2024-03-29T11:36:43.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.515+0000 7f0af7fff640 10 monclient: build_initial_monmap 2024-03-29T11:36:43.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.515+0000 7f0af7fff640 10 monclient: monmap: 2024-03-29T11:36:43.517 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:43.517 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:43.517 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:36:43.516574+0000 2024-03-29T11:36:43.517 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:36:43.516574+0000 2024-03-29T11:36:43.517 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:43.517 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:43.517 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:43.518 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:36:43.518 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:36:43.518 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:43.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.519+0000 7f0af7fff640 1 Processor -- start 2024-03-29T11:36:43.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.520+0000 7f0af7fff640 1 -- start start 2024-03-29T11:36:43.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.520+0000 7f0af7fff640 10 monclient: init 2024-03-29T11:36:43.520 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.520+0000 7f0af7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:43.520 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.520+0000 7f0af7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:36:43.520 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.520+0000 7f0af7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f0af0034850 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:43.520 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.520+0000 7f0af7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0af002a640 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:43.520 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.521+0000 7f0af7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0af00327d0 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:43.520 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.521+0000 7f0af7fff640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af002c820 con 0x7f0af00327d0 2024-03-29T11:36:43.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.521+0000 7f0af7fff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af002a120 con 0x7f0af0034850 2024-03-29T11:36:43.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.521+0000 7f0af7fff640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af00c75b0 con 0x7f0af002a640 2024-03-29T11:36:43.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.521+0000 7f0af7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:43.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.521+0000 7f0af7fff640 10 monclient(hunting): authenticate will time out at 992.829183s 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.524+0000 7f0af77fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f0af00327d0 0x7f0af00cf240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:40753/0 (socket says 172.21.15.145:40753) 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.524+0000 7f0af77fe640 1 -- 172.21.15.145:0/2022320025 learned_addr learned my addr 172.21.15.145:0/2022320025 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.525+0000 7f0af5ffb640 1 -- 172.21.15.145:0/2022320025 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2411094910 0 0) 0x7f0af002c820 con 0x7f0af00327d0 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.525+0000 7f0af5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.525+0000 7f0af5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.525+0000 7f0af5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.525+0000 7f0af5ffb640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.525+0000 7f0af5ffb640 1 -- 172.21.15.145:0/2022320025 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae0003520 con 0x7f0af00327d0 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.526+0000 7f0af5ffb640 1 -- 172.21.15.145:0/2022320025 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 648492713 0 0) 0x7f0ae0003520 con 0x7f0af00327d0 2024-03-29T11:36:43.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.526+0000 7f0af5ffb640 1 -- 172.21.15.145:0/2022320025 >> v1:172.21.15.145:6790/0 conn(0x7f0af002a640 legacy=0x7f0af00320c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 1 -- 172.21.15.145:0/2022320025 >> v1:172.21.15.203:6789/0 conn(0x7f0af0034850 legacy=0x7f0af00c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 1 monclient: found mon.noname-a 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 1 -- 172.21.15.145:0/2022320025 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af0033110 con 0x7f0af00327d0 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 1 -- 172.21.15.145:0/2022320025 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0ae8001690 con 0x7f0af00327d0 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af5ffb640 10 monclient: dump: 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:43.529 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af7fff640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:36:43.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.528+0000 7f0af7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:43.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.530+0000 7f0af5ffb640 1 -- 172.21.15.145:0/2022320025 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ae8001860 con 0x7f0af00327d0 2024-03-29T11:36:43.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.530+0000 7f0af5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:43.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.530+0000 7f0af5ffb640 1 -- 172.21.15.145:0/2022320025 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0ae8001cb0 con 0x7f0af00327d0 2024-03-29T11:36:43.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.530+0000 7f0af5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.530+0000 7f0af5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.530+0000 7f0af5ffb640 10 monclient: dump: 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.530+0000 7f0af7fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.530+0000 7f0af7fff640 4 set_mon_vals no callback set 2024-03-29T11:36:43.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.531+0000 7f0af7fff640 10 monclient: shutdown 2024-03-29T11:36:43.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.531+0000 7f0af7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:43.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.531+0000 7f0af7fff640 1 -- 172.21.15.145:0/2022320025 >> v1:172.21.15.145:6789/0 conn(0x7f0af00327d0 legacy=0x7f0af00cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.531+0000 7f0af7fff640 1 -- 172.21.15.145:0/2022320025 shutdown_connections 2024-03-29T11:36:43.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.531+0000 7f0af7fff640 1 -- 172.21.15.145:0/2022320025 >> 172.21.15.145:0/2022320025 conn(0x7f0af00058f0 msgr2=0x7f0af0005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:43.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.532+0000 7f0af7fff640 1 -- 172.21.15.145:0/2022320025 shutdown_connections 2024-03-29T11:36:43.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.532+0000 7f0af7fff640 1 -- 172.21.15.145:0/2022320025 wait complete. 2024-03-29T11:36:43.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.533+0000 7f0af7fff640 10 monclient: build_initial_monmap 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.533+0000 7f0af7fff640 10 monclient: monmap: 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:36:43.534 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:43.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.534+0000 7f0af7fff640 1 librados: starting msgr at 2024-03-29T11:36:43.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.534+0000 7f0af7fff640 1 librados: starting objecter 2024-03-29T11:36:43.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.535+0000 7f0af7fff640 1 Processor -- start 2024-03-29T11:36:43.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.535+0000 7f0af7fff640 1 -- start start 2024-03-29T11:36:43.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.535+0000 7f0af7fff640 1 librados: setting wanted keys 2024-03-29T11:36:43.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.536+0000 7f0af7fff640 1 librados: calling monclient init 2024-03-29T11:36:43.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.536+0000 7f0af7fff640 10 monclient: init 2024-03-29T11:36:43.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.536+0000 7f0af7fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:43.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.537+0000 7f0af7fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:43.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.539+0000 7f0af7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f0af002a640 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:43.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.539+0000 7f0af7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f0af00327d0 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:43.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.539+0000 7f0af7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f0af0034850 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:43.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.539+0000 7f0af7fff640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af0030750 con 0x7f0af00327d0 2024-03-29T11:36:43.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.539+0000 7f0af7fff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af0030a30 con 0x7f0af0034850 2024-03-29T11:36:43.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.539+0000 7f0af7fff640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0af002f0c0 con 0x7f0af002a640 2024-03-29T11:36:43.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.539+0000 7f0af7fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:43.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.539+0000 7f0af7fff640 10 monclient(hunting): authenticate will time out at 992.847993s 2024-03-29T11:36:43.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.541+0000 7f0af6ffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f0af002a640 0x7f0af015c950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:33501/0 (socket says 172.21.15.145:33501) 2024-03-29T11:36:43.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.541+0000 7f0af6ffd640 1 -- 172.21.15.145:0/1107112498 learned_addr learned my addr 172.21.15.145:0/1107112498 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.543+0000 7f0adf7fe640 10 client.?.objecter ms_handle_connect 0x7f0af002a640 2024-03-29T11:36:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.543+0000 7f0adf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.543+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115001491 0 0) 0x7f0af002f0c0 con 0x7f0af002a640 2024-03-29T11:36:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.543+0000 7f0adf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.543+0000 7f0adf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.543+0000 7f0adf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.543+0000 7f0adf7fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:36:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.543+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae4004eb0 con 0x7f0af002a640 2024-03-29T11:36:43.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.547+0000 7f0adf7fe640 10 client.?.objecter ms_handle_connect 0x7f0af00327d0 2024-03-29T11:36:43.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.547+0000 7f0adf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:43.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.547+0000 7f0adf7fe640 10 client.?.objecter ms_handle_connect 0x7f0af0034850 2024-03-29T11:36:43.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.547+0000 7f0adf7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:43.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.547+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3559850416 0 0) 0x7f0ae4004eb0 con 0x7f0af002a640 2024-03-29T11:36:43.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0af002f0c0 con 0x7f0af002a640 2024-03-29T11:36:43.557 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830716506 0 0) 0x7f0af0030750 con 0x7f0af00327d0 2024-03-29T11:36:43.557 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:43.557 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:43.557 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:43.557 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:36:43.557 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ae4004eb0 con 0x7f0af00327d0 2024-03-29T11:36:43.558 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871985690 0 0) 0x7f0af0030a30 con 0x7f0af0034850 2024-03-29T11:36:43.558 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:43.558 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:43.558 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:43.558 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.548+0000 7f0adf7fe640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:36:43.558 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0af0030750 con 0x7f0af0034850 2024-03-29T11:36:43.558 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0af804fdd0 con 0x7f0af002a640 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 10 monclient(hunting): dump: 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:43.559 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:43.560 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:43.560 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:43.560 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:43.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:43.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2906874359 0 0) 0x7f0af002f0c0 con 0x7f0af002a640 2024-03-29T11:36:43.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 >> v1:172.21.15.203:6789/0 conn(0x7f0af0034850 legacy=0x7f0af0031560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 >> v1:172.21.15.145:6789/0 conn(0x7f0af00327d0 legacy=0x7f0af0030e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:43.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 1 monclient: found mon.c 2024-03-29T11:36:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af0031c70 con 0x7f0af002a640 2024-03-29T11:36:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0adf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0af7fff640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:36:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0af7fff640 10 monclient: _renew_subs 2024-03-29T11:36:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0af7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af002ece0 con 0x7f0af002a640 2024-03-29T11:36:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0af7fff640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0af7fff640 10 monclient: _renew_subs 2024-03-29T11:36:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0af7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af002e880 con 0x7f0af002a640 2024-03-29T11:36:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.549+0000 7f0af7fff640 1 librados: init done 2024-03-29T11:36:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.550+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0af804f400 con 0x7f0af002a640 2024-03-29T11:36:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.550+0000 7f0adf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.550+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0af8075f10 con 0x7f0af002a640 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.550+0000 7f0af4ff9640 4 set_mon_vals no callback set 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.550+0000 7f0adf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.550+0000 7f0adf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.551+0000 7f0adf7fe640 10 monclient: dump: 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:43.563 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:43.564 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:43.564 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:43.564 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:43.564 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.551+0000 7f0af7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.552+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f0af8067bf0 con 0x7f0af002a640 2024-03-29T11:36:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0af67fc640 10 monclient: get_auth_request con 0x7f0ae406ddf0 auth_method 0 2024-03-29T11:36:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f0af80ff3d0 con 0x7f0af002a640 2024-03-29T11:36:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0adf7fe640 10 client.4298.objecter ms_dispatch 0x7f0af8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:43.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0adf7fe640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:43.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0adf7fe640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:43.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0adf7fe640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:36:43.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0af7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:43.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0af7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:43.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ad8005080 con 0x7f0af002a640 2024-03-29T11:36:43.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.553+0000 7f0adf7fe640 10 client.4298.objecter ms_handle_connect 0x7f0ae406ddf0 2024-03-29T11:36:43.566 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.564+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0af80cacb0 con 0x7f0af002a640 2024-03-29T11:36:43.566 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.564+0000 7f0adf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:43.566 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.564+0000 7f0adf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:43.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.889+0000 7f0af7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:43.888 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.889+0000 7f0af7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:43.888 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.889+0000 7f0af7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:43.888 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.889+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ad8005330 con 0x7f0af002a640 2024-03-29T11:36:43.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.894+0000 7f0adf7fe640 1 -- 172.21.15.145:0/1107112498 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f0af80ccbe0 con 0x7f0af002a640 2024-03-29T11:36:43.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.894+0000 7f0adf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:43.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.894+0000 7f0adf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:43.894 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:36:43.894 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:43.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.897+0000 7f0af7fff640 10 librados: watch_flush enter 2024-03-29T11:36:43.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 10 librados: watch_flush exit 2024-03-29T11:36:43.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:36:43.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:36:43.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 >> v1:172.21.15.145:6816/227372239 conn(0x7f0ae406ddf0 legacy=0x7f0ae40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 10 monclient: shutdown 2024-03-29T11:36:43.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:43.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 >> v1:172.21.15.145:6790/0 conn(0x7f0af002a640 legacy=0x7f0af015c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:43.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 shutdown_connections 2024-03-29T11:36:43.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 >> 172.21.15.145:0/1107112498 conn(0x7f0af00058f0 msgr2=0x7f0af0029610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:43.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 shutdown_connections 2024-03-29T11:36:43.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.902+0000 7f0af7fff640 1 -- 172.21.15.145:0/1107112498 wait complete. 2024-03-29T11:36:43.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:43.906+0000 7f0af7fff640 1 librados: shutdown 2024-03-29T11:36:43.923 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:50.729 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:50.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.887+0000 7f1a816c9640 10 monclient: get_monmap_and_config 2024-03-29T11:36:50.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.887+0000 7f1a816c9640 10 monclient: build_initial_monmap 2024-03-29T11:36:50.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.888+0000 7f1a816c9640 10 monclient: monmap: 2024-03-29T11:36:50.887 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:50.887 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:50.887 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:36:50.888271+0000 2024-03-29T11:36:50.887 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:36:50.888271+0000 2024-03-29T11:36:50.888 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:50.888 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:50.888 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:50.888 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:36:50.888 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:36:50.888 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:50.888 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.890+0000 7f1a816c9640 1 Processor -- start 2024-03-29T11:36:50.889 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.891+0000 7f1a816c9640 1 -- start start 2024-03-29T11:36:50.889 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.891+0000 7f1a816c9640 10 monclient: init 2024-03-29T11:36:50.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.892+0000 7f1a816c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:50.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.892+0000 7f1a816c9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:36:50.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.892+0000 7f1a816c9640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7c06ade0 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:50.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.892+0000 7f1a816c9640 10 monclient(hunting): picked mon.noname-c con 0x7f1a7c0afa30 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:50.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.896+0000 7f1a7affd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f1a7c06ade0 0x7f1a7c071d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:60173/0 (socket says 172.21.15.145:60173) 2024-03-29T11:36:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.897+0000 7f1a816c9640 10 monclient(hunting): picked mon.noname-a con 0x7f1a7c1433e0 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.897+0000 7f1a816c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c072670 con 0x7f1a7c1433e0 2024-03-29T11:36:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.897+0000 7f1a816c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c13da60 con 0x7f1a7c0afa30 2024-03-29T11:36:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.897+0000 7f1a816c9640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c143fa0 con 0x7f1a7c06ade0 2024-03-29T11:36:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.897+0000 7f1a816c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.897+0000 7f1a816c9640 10 monclient(hunting): authenticate will time out at 1000.204796s 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.897+0000 7f1a7affd640 1 -- 172.21.15.145:0/922393795 learned_addr learned my addr 172.21.15.145:0/922393795 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3322701422 0 0) 0x7f1a7c143fa0 con 0x7f1a7c06ade0 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a68003520 con 0x7f1a7c06ade0 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688562600 0 0) 0x7f1a7c13da60 con 0x7f1a7c0afa30 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:36:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c143fa0 con 0x7f1a7c0afa30 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 674367798 0 0) 0x7f1a7c143fa0 con 0x7f1a7c0afa30 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 >> v1:172.21.15.145:6790/0 conn(0x7f1a7c06ade0 legacy=0x7f1a7c071d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 >> v1:172.21.15.145:6789/0 conn(0x7f1a7c1433e0 legacy=0x7f1a7c143890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c14f730 con 0x7f1a7c0afa30 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1a6c001520 con 0x7f1a7c0afa30 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a7a7fc640 10 monclient: dump: 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a816c9640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.898+0000 7f1a816c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.899+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a6c001700 con 0x7f1a7c0afa30 2024-03-29T11:36:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.899+0000 7f1a7a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.899+0000 7f1a7a7fc640 1 -- 172.21.15.145:0/922393795 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1a6c001a90 con 0x7f1a7c0afa30 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.899+0000 7f1a7a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.899+0000 7f1a7a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.899+0000 7f1a7a7fc640 10 monclient: dump: 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:50.904 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:50.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.899+0000 7f1a816c9640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:50.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.899+0000 7f1a816c9640 4 set_mon_vals no callback set 2024-03-29T11:36:50.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.900+0000 7f1a816c9640 10 monclient: shutdown 2024-03-29T11:36:50.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.900+0000 7f1a816c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:50.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.900+0000 7f1a816c9640 1 -- 172.21.15.145:0/922393795 >> v1:172.21.15.203:6789/0 conn(0x7f1a7c0afa30 legacy=0x7f1a7c13d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.903+0000 7f1a816c9640 1 -- 172.21.15.145:0/922393795 shutdown_connections 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.903+0000 7f1a816c9640 1 -- 172.21.15.145:0/922393795 >> 172.21.15.145:0/922393795 conn(0x7f1a7c067470 msgr2=0x7f1a7c0b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.904+0000 7f1a816c9640 1 -- 172.21.15.145:0/922393795 shutdown_connections 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.906+0000 7f1a816c9640 1 -- 172.21.15.145:0/922393795 wait complete. 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.907+0000 7f1a816c9640 10 monclient: build_initial_monmap 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.907+0000 7f1a816c9640 10 monclient: monmap: 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:36:50.906 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:50.907 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:50.907 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:50.907 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:36:50.907 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:36:50.907 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:50.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.908+0000 7f1a816c9640 1 librados: starting msgr at 2024-03-29T11:36:50.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.908+0000 7f1a816c9640 1 librados: starting objecter 2024-03-29T11:36:50.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.909+0000 7f1a816c9640 1 Processor -- start 2024-03-29T11:36:50.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.911+0000 7f1a816c9640 1 -- start start 2024-03-29T11:36:50.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.911+0000 7f1a816c9640 1 librados: setting wanted keys 2024-03-29T11:36:50.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.911+0000 7f1a816c9640 1 librados: calling monclient init 2024-03-29T11:36:50.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.911+0000 7f1a816c9640 10 monclient: init 2024-03-29T11:36:50.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.912+0000 7f1a816c9640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:50.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.912+0000 7f1a816c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:36:50.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.912+0000 7f1a816c9640 10 monclient(hunting): picked mon.noname-b con 0x7f1a7c06ade0 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:50.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.913+0000 7f1a816c9640 10 monclient(hunting): picked mon.noname-a con 0x7f1a7c0afa30 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:50.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.913+0000 7f1a816c9640 10 monclient(hunting): picked mon.noname-c con 0x7f1a7c1433e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:50.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.913+0000 7f1a816c9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c13f670 con 0x7f1a7c0afa30 2024-03-29T11:36:50.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.913+0000 7f1a816c9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c1407d0 con 0x7f1a7c06ade0 2024-03-29T11:36:50.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.913+0000 7f1a816c9640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a7c13e100 con 0x7f1a7c1433e0 2024-03-29T11:36:50.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.913+0000 7f1a816c9640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:50.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.913+0000 7f1a816c9640 10 monclient(hunting): authenticate will time out at 1000.220083s 2024-03-29T11:36:50.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.917+0000 7f1a7b7fe640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f1a7c1433e0 0x7f1a7c1dd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:60329/0 (socket says 172.21.15.145:60329) 2024-03-29T11:36:50.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.917+0000 7f1a7b7fe640 1 -- 172.21.15.145:0/2422452818 learned_addr learned my addr 172.21.15.145:0/2422452818 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:50.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 client.?.objecter ms_handle_connect 0x7f1a7c1433e0 2024-03-29T11:36:50.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:50.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 client.?.objecter ms_handle_connect 0x7f1a7c06ade0 2024-03-29T11:36:50.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:50.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 client.?.objecter ms_handle_connect 0x7f1a7c0afa30 2024-03-29T11:36:50.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:50.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597784253 0 0) 0x7f1a7c13e100 con 0x7f1a7c1433e0 2024-03-29T11:36:50.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:50.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:50.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:50.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:36:50.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a5c004eb0 con 0x7f1a7c1433e0 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639467984 0 0) 0x7f1a7c1407d0 con 0x7f1a7c06ade0 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): _negotiate 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a7c13e100 con 0x7f1a7c06ade0 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3124021528 0 0) 0x7f1a5c004eb0 con 0x7f1a7c1433e0 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a7c1407d0 con 0x7f1a7c1433e0 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1a74001c30 con 0x7f1a7c1433e0 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): dump: 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:50.928 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2984714320 0 0) 0x7f1a7c1407d0 con 0x7f1a7c1433e0 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 >> v1:172.21.15.203:6789/0 conn(0x7f1a7c06ade0 legacy=0x7f1a7c13ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 >> v1:172.21.15.145:6789/0 conn(0x7f1a7c0afa30 legacy=0x7f1a7c12abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient: _finish_hunting 0 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 monclient: found mon.c 2024-03-29T11:36:50.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7c13dc70 con 0x7f1a7c1433e0 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient: _finish_auth 0 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 10 monclient: _check_auth_tickets 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.921+0000 7f1a816c9640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.922+0000 7f1a816c9640 10 monclient: _renew_subs 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.922+0000 7f1a816c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.922+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a7c140bf0 con 0x7f1a7c1433e0 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.922+0000 7f1a816c9640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.922+0000 7f1a816c9640 10 monclient: _renew_subs 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.922+0000 7f1a816c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.922+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a7c12b4a0 con 0x7f1a7c1433e0 2024-03-29T11:36:50.930 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.922+0000 7f1a816c9640 1 librados: init done 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.924+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a74003970 con 0x7f1a7c1433e0 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.924+0000 7f1a73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.924+0000 7f1a79ffb640 4 set_mon_vals no callback set 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1a74003d80 con 0x7f1a7c1433e0 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a73fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a73fff640 10 monclient: dump: 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:50.931 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f1a74003fa0 con 0x7f1a7c1433e0 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f1a74084d70 con 0x7f1a7c1433e0 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a73fff640 10 client.4304.objecter ms_dispatch 0x7f1a7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a7affd640 10 monclient: get_auth_request con 0x7f1a5c0766e0 auth_method 0 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a73fff640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.925+0000 7f1a73fff640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.926+0000 7f1a73fff640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.926+0000 7f1a73fff640 10 client.4304.objecter ms_handle_connect 0x7f1a5c0766e0 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.927+0000 7f1a816c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.927+0000 7f1a816c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.927+0000 7f1a816c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:50.932 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.927+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a60005080 con 0x7f1a7c1433e0 2024-03-29T11:36:50.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.938+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a740506d0 con 0x7f1a7c1433e0 2024-03-29T11:36:50.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.938+0000 7f1a73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:50.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:50.938+0000 7f1a73fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:51.259 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.261+0000 7f1a816c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:51.259 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.261+0000 7f1a816c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:51.259 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.261+0000 7f1a816c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:36:51.260 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.261+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a60005330 con 0x7f1a7c1433e0 2024-03-29T11:36:51.260 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.261+0000 7f1a73fff640 1 -- 172.21.15.145:0/2422452818 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f1a74052600 con 0x7f1a7c1433e0 2024-03-29T11:36:51.260 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.261+0000 7f1a73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:51.260 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.261+0000 7f1a73fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:51.260 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:36:51.260 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:51.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 10 librados: watch_flush enter 2024-03-29T11:36:51.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 10 librados: watch_flush exit 2024-03-29T11:36:51.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T11:36:51.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 10 client.4304.objecter successfully canceled tick 2024-03-29T11:36:51.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 >> v1:172.21.15.145:6816/227372239 conn(0x7f1a5c0766e0 legacy=0x7f1a5c078ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:51.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 10 monclient: shutdown 2024-03-29T11:36:51.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:51.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 >> v1:172.21.15.145:6790/0 conn(0x7f1a7c1433e0 legacy=0x7f1a7c1dd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:51.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 shutdown_connections 2024-03-29T11:36:51.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.265+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 >> 172.21.15.145:0/2422452818 conn(0x7f1a7c067470 msgr2=0x7f1a7c0b3120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:51.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.266+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 shutdown_connections 2024-03-29T11:36:51.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.266+0000 7f1a816c9640 1 -- 172.21.15.145:0/2422452818 wait complete. 2024-03-29T11:36:51.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:51.267+0000 7f1a816c9640 1 librados: shutdown 2024-03-29T11:36:51.284 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:36:58.090 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:36:58.237 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.237+0000 7faed2fd7640 10 monclient: get_monmap_and_config 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.237+0000 7faed2fd7640 10 monclient: build_initial_monmap 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.237+0000 7faed2fd7640 10 monclient: monmap: 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:36:58.238336+0000 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:36:58.238336+0000 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:36:58.238 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:58.241 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.239+0000 7faed2fd7640 1 Processor -- start 2024-03-29T11:36:58.241 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.239+0000 7faed2fd7640 1 -- start start 2024-03-29T11:36:58.242 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.239+0000 7faed2fd7640 10 monclient: init 2024-03-29T11:36:58.242 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:58.242 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:36:58.242 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 10 monclient(hunting): picked mon.noname-b con 0x7faecc0a9930 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:58.242 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 10 monclient(hunting): picked mon.noname-a con 0x7faecc149dc0 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:58.242 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 10 monclient(hunting): picked mon.noname-c con 0x7faecc14ff40 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:58.242 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0aae60 con 0x7faecc149dc0 2024-03-29T11:36:58.242 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0b4310 con 0x7faecc14ff40 2024-03-29T11:36:58.242 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0afdd0 con 0x7faecc0a9930 2024-03-29T11:36:58.243 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:58.243 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.240+0000 7faed2fd7640 10 monclient(hunting): authenticate will time out at 1007.548547s 2024-03-29T11:36:58.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.244+0000 7faed0d73640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7faecc0a9930 0x7faecc143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:57951/0 (socket says 172.21.15.145:57951) 2024-03-29T11:36:58.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.244+0000 7faed0d73640 1 -- 172.21.15.145:0/1799293206 learned_addr learned my addr 172.21.15.145:0/1799293206 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:58.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.246+0000 7faecb7fe640 1 -- 172.21.15.145:0/1799293206 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376283450 0 0) 0x7faecc0afdd0 con 0x7faecc0a9930 2024-03-29T11:36:58.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.246+0000 7faecb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.246+0000 7faecb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.246+0000 7faecb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.246+0000 7faecb7fe640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:36:58.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.246+0000 7faecb7fe640 1 -- 172.21.15.145:0/1799293206 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb4003520 con 0x7faecc0a9930 2024-03-29T11:36:58.247 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 1 -- 172.21.15.145:0/1799293206 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2290561534 0 0) 0x7faeb4003520 con 0x7faecc0a9930 2024-03-29T11:36:58.247 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 1 -- 172.21.15.145:0/1799293206 >> v1:172.21.15.203:6789/0 conn(0x7faecc14ff40 legacy=0x7faecc152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.247 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 1 -- 172.21.15.145:0/1799293206 >> v1:172.21.15.145:6789/0 conn(0x7faecc149dc0 legacy=0x7faecc14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:36:58.248 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 1 -- 172.21.15.145:0/1799293206 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc14a950 con 0x7faecc0a9930 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faed2fd7640 5 monclient: authenticate success, global_id 4307 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faed2fd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 1 -- 172.21.15.145:0/1799293206 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faec0001690 con 0x7faecc0a9930 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 10 monclient: dump: 2024-03-29T11:36:58.249 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faed2fd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 1 -- 172.21.15.145:0/1799293206 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faec0001920 con 0x7faecc0a9930 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faecb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faed2fd7640 10 monclient: get_monmap_and_config success 2024-03-29T11:36:58.250 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.247+0000 7faed2fd7640 4 set_mon_vals no callback set 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.248+0000 7faecb7fe640 1 -- 172.21.15.145:0/1799293206 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faec0001cb0 con 0x7faecc0a9930 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.248+0000 7faecb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.248+0000 7faecb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.248+0000 7faecb7fe640 10 monclient: dump: 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:58.251 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:58.252 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:58.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.248+0000 7faed2fd7640 10 monclient: shutdown 2024-03-29T11:36:58.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.248+0000 7faed2fd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:58.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.248+0000 7faed2fd7640 1 -- 172.21.15.145:0/1799293206 >> v1:172.21.15.145:6790/0 conn(0x7faecc0a9930 legacy=0x7faecc143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.248+0000 7faed2fd7640 1 -- 172.21.15.145:0/1799293206 shutdown_connections 2024-03-29T11:36:58.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.248+0000 7faed2fd7640 1 -- 172.21.15.145:0/1799293206 >> 172.21.15.145:0/1799293206 conn(0x7faecc0b35e0 msgr2=0x7faecc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.249+0000 7faed2fd7640 1 -- 172.21.15.145:0/1799293206 shutdown_connections 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.249+0000 7faed2fd7640 1 -- 172.21.15.145:0/1799293206 wait complete. 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.250+0000 7faed2fd7640 10 monclient: build_initial_monmap 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.250+0000 7faed2fd7640 10 monclient: monmap: 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:36:58.253 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.250+0000 7faed2fd7640 1 librados: starting msgr at 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.250+0000 7faed2fd7640 1 librados: starting objecter 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.250+0000 7faed2fd7640 1 Processor -- start 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.250+0000 7faed2fd7640 1 -- start start 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.250+0000 7faed2fd7640 1 librados: setting wanted keys 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.250+0000 7faed2fd7640 1 librados: calling monclient init 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.250+0000 7faed2fd7640 10 monclient: init 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.251+0000 7faed2fd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.251+0000 7faed2fd7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:36:58.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.255+0000 7faed2fd7640 10 monclient(hunting): picked mon.noname-a con 0x7faecc0a9930 addr v1:172.21.15.145:6789/0 2024-03-29T11:36:58.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.255+0000 7faed2fd7640 10 monclient(hunting): picked mon.noname-c con 0x7faecc149dc0 addr v1:172.21.15.145:6790/0 2024-03-29T11:36:58.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.255+0000 7faed2fd7640 10 monclient(hunting): picked mon.noname-b con 0x7faecc14ff40 addr v1:172.21.15.203:6789/0 2024-03-29T11:36:58.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.255+0000 7faed2fd7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc11e9f0 con 0x7faecc0a9930 2024-03-29T11:36:58.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.255+0000 7faed2fd7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc13d840 con 0x7faecc14ff40 2024-03-29T11:36:58.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.255+0000 7faed2fd7640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faecc0714a0 con 0x7faecc149dc0 2024-03-29T11:36:58.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.255+0000 7faed2fd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:36:58.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.255+0000 7faed2fd7640 10 monclient(hunting): authenticate will time out at 1007.563291s 2024-03-29T11:36:58.260 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.258+0000 7faed0d73640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7faecc0a9930 0x7faecc06fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:60303/0 (socket says 172.21.15.145:60303) 2024-03-29T11:36:58.260 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.258+0000 7faed0d73640 1 -- 172.21.15.145:0/2382273117 learned_addr learned my addr 172.21.15.145:0/2382273117 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:36:58.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 client.?.objecter ms_handle_connect 0x7faecc0a9930 2024-03-29T11:36:58.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:58.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626313041 0 0) 0x7faecc11e9f0 con 0x7faecc0a9930 2024-03-29T11:36:58.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 monclient(hunting): my global_id is 4299 2024-03-29T11:36:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeb8004eb0 con 0x7faecc0a9930 2024-03-29T11:36:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 client.?.objecter ms_handle_connect 0x7faecc14ff40 2024-03-29T11:36:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 client.?.objecter ms_handle_connect 0x7faecc149dc0 2024-03-29T11:36:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.261+0000 7faec97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417682666 0 0) 0x7faecc13d840 con 0x7faecc14ff40 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faecc11e9f0 con 0x7faecc14ff40 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981541367 0 0) 0x7faecc0714a0 con 0x7faecc149dc0 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:36:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faecc13d840 con 0x7faecc149dc0 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2226998000 0 0) 0x7faeb8004eb0 con 0x7faecc0a9930 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faecc0714a0 con 0x7faecc0a9930 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faec0001550 con 0x7faecc0a9930 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): dump: 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:58.269 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3479369230 0 0) 0x7faecc11e9f0 con 0x7faecc14ff40 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.262+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faeb8004eb0 con 0x7faecc14ff40 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2970909944 0 0) 0x7faecc13d840 con 0x7faecc149dc0 2024-03-29T11:36:58.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faecc11e9f0 con 0x7faecc149dc0 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faebc001790 con 0x7faecc14ff40 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 10 monclient(hunting): dump: 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:58.271 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faec4001aa0 con 0x7faecc149dc0 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 10 monclient(hunting): dump: 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:58.272 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3093604253 0 0) 0x7faeb8004eb0 con 0x7faecc14ff40 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.263+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 >> v1:172.21.15.145:6790/0 conn(0x7faecc149dc0 legacy=0x7faecc13c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 >> v1:172.21.15.145:6789/0 conn(0x7faecc0a9930 legacy=0x7faecc06fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faec97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faec97fa640 1 monclient: found mon.b 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faec97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faec97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faecc13d360 con 0x7faecc14ff40 2024-03-29T11:36:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faec97fa640 10 monclient: _finish_auth 0 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faec97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faec97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faed2fd7640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faed2fd7640 10 monclient: _renew_subs 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faed2fd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faecc071770 con 0x7faecc14ff40 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faed2fd7640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faed2fd7640 10 monclient: _renew_subs 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faed2fd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faecc06f390 con 0x7faecc14ff40 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.264+0000 7faed2fd7640 1 librados: init done 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.265+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faebc0018f0 con 0x7faecc14ff40 2024-03-29T11:36:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.265+0000 7faec97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.265+0000 7faeca7fc640 4 set_mon_vals no callback set 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.266+0000 7faed2fd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.267+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faebc0035f0 con 0x7faecc14ff40 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.267+0000 7faec97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.267+0000 7faec97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.267+0000 7faec97fa640 10 monclient: dump: 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:36:58.275 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.267+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7faebc003810 con 0x7faecc14ff40 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.268+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7faebc083730 con 0x7faecc14ff40 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.268+0000 7faec97fa640 10 client.4300.objecter ms_dispatch 0x7faecc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.268+0000 7faec97fa640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.268+0000 7faec97fa640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.268+0000 7faec97fa640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-29T11:36:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.268+0000 7faed2fd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:58.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.268+0000 7faed2fd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:58.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.268+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeac005080 con 0x7faecc14ff40 2024-03-29T11:36:58.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.268+0000 7faed0d73640 10 monclient: get_auth_request con 0x7faeb806e160 auth_method 0 2024-03-29T11:36:58.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.269+0000 7faec97fa640 10 client.4300.objecter ms_handle_connect 0x7faeb806e160 2024-03-29T11:36:58.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.276+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faebc04f090 con 0x7faecc14ff40 2024-03-29T11:36:58.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.276+0000 7faec97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:36:58.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.276+0000 7faec97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:36:58.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.598+0000 7faed2fd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:58.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.598+0000 7faed2fd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:58.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.598+0000 7faed2fd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:36:58.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.598+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faeac005330 con 0x7faecc14ff40 2024-03-29T11:36:58.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.599+0000 7faec97fa640 1 -- 172.21.15.145:0/2382273117 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7faebc050fc0 con 0x7faecc14ff40 2024-03-29T11:36:58.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.599+0000 7faec97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:36:58.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.599+0000 7faec97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:36:58.599 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:36:58.599 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:36:58.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.603+0000 7faed2fd7640 10 librados: watch_flush enter 2024-03-29T11:36:58.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.603+0000 7faed2fd7640 10 librados: watch_flush exit 2024-03-29T11:36:58.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.603+0000 7faed2fd7640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-29T11:36:58.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.603+0000 7faed2fd7640 10 client.4300.objecter successfully canceled tick 2024-03-29T11:36:58.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.603+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 >> v1:172.21.15.145:6816/227372239 conn(0x7faeb806e160 legacy=0x7faeb8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.603+0000 7faed2fd7640 10 monclient: shutdown 2024-03-29T11:36:58.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.603+0000 7faed2fd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:36:58.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.603+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 >> v1:172.21.15.203:6789/0 conn(0x7faecc14ff40 legacy=0x7faecc13c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:36:58.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.604+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 shutdown_connections 2024-03-29T11:36:58.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.604+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 >> 172.21.15.145:0/2382273117 conn(0x7faecc0b35e0 msgr2=0x7faecc069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:36:58.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.604+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 shutdown_connections 2024-03-29T11:36:58.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.604+0000 7faed2fd7640 1 -- 172.21.15.145:0/2382273117 wait complete. 2024-03-29T11:36:58.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:36:58.604+0000 7faed2fd7640 1 librados: shutdown 2024-03-29T11:36:58.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:05.428 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.584+0000 7f396359e640 10 monclient: get_monmap_and_config 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.584+0000 7f396359e640 10 monclient: build_initial_monmap 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.584+0000 7f396359e640 10 monclient: monmap: 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:37:05.585456+0000 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:37:05.585456+0000 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:37:05.584 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:37:05.585 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:05.585 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.585+0000 7f396359e640 1 Processor -- start 2024-03-29T11:37:05.585 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.586+0000 7f396359e640 1 -- start start 2024-03-29T11:37:05.585 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.586+0000 7f396359e640 10 monclient: init 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 10 monclient(hunting): picked mon.noname-b con 0x7f39640b4920 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 10 monclient(hunting): picked mon.noname-c con 0x7f39640b4200 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 10 monclient(hunting): picked mon.noname-a con 0x7f396413f9a0 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39640b0930 con 0x7f396413f9a0 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39640af460 con 0x7f39640b4200 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39640aeda0 con 0x7f39640b4920 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.587+0000 7f396359e640 10 monclient(hunting): authenticate will time out at 1014.895183s 2024-03-29T11:37:05.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.590+0000 7f396259c640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f39640b4920 0x7f396413f290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:50495/0 (socket says 172.21.15.145:50495) 2024-03-29T11:37:05.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.590+0000 7f396259c640 1 -- 172.21.15.145:0/3544515939 learned_addr learned my addr 172.21.15.145:0/3544515939 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:05.594 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.591+0000 7f396159a640 1 -- 172.21.15.145:0/3544515939 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895695213 0 0) 0x7f39640aeda0 con 0x7f39640b4920 2024-03-29T11:37:05.594 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.591+0000 7f396159a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:05.594 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.591+0000 7f396159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.591+0000 7f396159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.591+0000 7f396159a640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.591+0000 7f396159a640 1 -- 172.21.15.145:0/3544515939 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f394c003520 con 0x7f39640b4920 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 1 -- 172.21.15.145:0/3544515939 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3043310346 0 0) 0x7f394c003520 con 0x7f39640b4920 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 1 -- 172.21.15.145:0/3544515939 >> v1:172.21.15.203:6789/0 conn(0x7f39640b4200 legacy=0x7f39640b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 1 -- 172.21.15.145:0/3544515939 >> v1:172.21.15.145:6789/0 conn(0x7f396413f9a0 legacy=0x7f396414d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: _finish_hunting 0 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 1 monclient: found mon.noname-b 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 1 -- 172.21.15.145:0/3544515939 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39640af980 con 0x7f39640b4920 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: _finish_auth 0 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: _check_auth_tickets 2024-03-29T11:37:05.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 1 -- 172.21.15.145:0/3544515939 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3958001690 con 0x7f39640b4920 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: dump: 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:05.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396359e640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 1 -- 172.21.15.145:0/3544515939 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3958001860 con 0x7f39640b4920 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 1 -- 172.21.15.145:0/3544515939 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3958001c70 con 0x7f39640b4920 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396159a640 10 monclient: dump: 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:05.597 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:05.598 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:05.598 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:05.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:05.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.592+0000 7f396359e640 4 set_mon_vals no callback set 2024-03-29T11:37:05.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.593+0000 7f396359e640 10 monclient: shutdown 2024-03-29T11:37:05.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.593+0000 7f396359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:05.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.593+0000 7f396359e640 1 -- 172.21.15.145:0/3544515939 >> v1:172.21.15.145:6790/0 conn(0x7f39640b4920 legacy=0x7f396413f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:05.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.598+0000 7f396359e640 1 -- 172.21.15.145:0/3544515939 shutdown_connections 2024-03-29T11:37:05.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.598+0000 7f396359e640 1 -- 172.21.15.145:0/3544515939 >> 172.21.15.145:0/3544515939 conn(0x7f396406d230 msgr2=0x7f3964066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:05.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.599+0000 7f396359e640 1 -- 172.21.15.145:0/3544515939 shutdown_connections 2024-03-29T11:37:05.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.599+0000 7f396359e640 1 -- 172.21.15.145:0/3544515939 wait complete. 2024-03-29T11:37:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.600+0000 7f396359e640 10 monclient: build_initial_monmap 2024-03-29T11:37:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.600+0000 7f396359e640 10 monclient: monmap: 2024-03-29T11:37:05.600 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:05.600 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:05.600 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:37:05.600 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:37:05.601 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:05.601 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:05.601 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:05.601 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:37:05.601 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:37:05.601 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.601+0000 7f396359e640 1 librados: starting msgr at 2024-03-29T11:37:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.601+0000 7f396359e640 1 librados: starting objecter 2024-03-29T11:37:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.602+0000 7f396359e640 1 Processor -- start 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.602+0000 7f396359e640 1 -- start start 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.602+0000 7f396359e640 1 librados: setting wanted keys 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.602+0000 7f396359e640 1 librados: calling monclient init 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.602+0000 7f396359e640 10 monclient: init 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 10 monclient(hunting): picked mon.noname-a con 0x7f39640b4200 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 10 monclient(hunting): picked mon.noname-c con 0x7f39640b4920 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 10 monclient(hunting): picked mon.noname-b con 0x7f396413f9a0 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3964145ff0 con 0x7f39640b4200 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396406e730 con 0x7f396413f9a0 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396406ea10 con 0x7f39640b4920 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.603+0000 7f396359e640 10 monclient(hunting): authenticate will time out at 1014.911150s 2024-03-29T11:37:05.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.606+0000 7f3962d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f396413f9a0 0x7f39641e6ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:45455/0 (socket says 172.21.15.145:45455) 2024-03-29T11:37:05.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.607+0000 7f3962d9d640 1 -- 172.21.15.145:0/3550567790 learned_addr learned my addr 172.21.15.145:0/3550567790 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.607+0000 7f3952ffd640 10 client.?.objecter ms_handle_connect 0x7f396413f9a0 2024-03-29T11:37:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.607+0000 7f3952ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.607+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120988853 0 0) 0x7f396406e730 con 0x7f396413f9a0 2024-03-29T11:37:05.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.607+0000 7f3952ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:05.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.607+0000 7f3952ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:05.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.608+0000 7f3952ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:05.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.608+0000 7f3952ffd640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:37:05.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.608+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3944004eb0 con 0x7f396413f9a0 2024-03-29T11:37:05.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.609+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4028417307 0 0) 0x7f3944004eb0 con 0x7f396413f9a0 2024-03-29T11:37:05.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.609+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f396406e730 con 0x7f396413f9a0 2024-03-29T11:37:05.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.609+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f395c001c60 con 0x7f396413f9a0 2024-03-29T11:37:05.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.609+0000 7f3952ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:05.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.609+0000 7f3952ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:05.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.609+0000 7f3952ffd640 10 monclient(hunting): dump: 2024-03-29T11:37:05.609 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:05.609 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:05.609 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:05.610 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:05.610 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:05.610 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:05.610 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:05.610 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:05.610 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:05.610 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:05.610 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.609+0000 7f3952ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:05.611 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.610+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1356714719 0 0) 0x7f396406e730 con 0x7f396413f9a0 2024-03-29T11:37:05.611 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.610+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 >> v1:172.21.15.145:6790/0 conn(0x7f39640b4920 legacy=0x7f39641de960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:05.616 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 >> v1:172.21.15.145:6789/0 conn(0x7f39640b4200 legacy=0x7f39641db940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:37:05.616 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f3952ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:05.616 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f3952ffd640 1 monclient: found mon.b 2024-03-29T11:37:05.616 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f3952ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:05.616 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f3952ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:05.616 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964145ad0 con 0x7f396413f9a0 2024-03-29T11:37:05.616 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f3952ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f3952ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f3952ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f396359e640 5 monclient: authenticate success, global_id 4303 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f396359e640 10 monclient: _renew_subs 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f396359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f396359e640 1 -- 172.21.15.145:0/3550567790 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39641385d0 con 0x7f396413f9a0 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.611+0000 7f396359e640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.612+0000 7f396359e640 10 monclient: _renew_subs 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.612+0000 7f396359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.612+0000 7f396359e640 1 -- 172.21.15.145:0/3550567790 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3964145820 con 0x7f396413f9a0 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.612+0000 7f396359e640 1 librados: init done 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.613+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f395c003990 con 0x7f396413f9a0 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.613+0000 7f3952ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.613+0000 7f3953fff640 4 set_mon_vals no callback set 2024-03-29T11:37:05.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.614+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f395c003da0 con 0x7f396413f9a0 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.614+0000 7f3952ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.614+0000 7f3952ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.614+0000 7f3952ffd640 10 monclient: dump: 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:05.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.616+0000 7f3950ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:05.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.616+0000 7f3950ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:05.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.616+0000 7f3950ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:05.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.616+0000 7f3950ff9640 1 -- 172.21.15.145:0/3550567790 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3948005080 con 0x7f396413f9a0 2024-03-29T11:37:05.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.617+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f395c003f00 con 0x7f396413f9a0 2024-03-29T11:37:05.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.617+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f395c084e90 con 0x7f396413f9a0 2024-03-29T11:37:05.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.617+0000 7f3952ffd640 10 client.4303.objecter ms_dispatch 0x7f3964000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:05.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.617+0000 7f3952ffd640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:05.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.617+0000 7f3952ffd640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:05.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.617+0000 7f3952ffd640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-29T11:37:05.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.620+0000 7f396259c640 10 monclient: get_auth_request con 0x7f394406dc80 auth_method 0 2024-03-29T11:37:05.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.621+0000 7f3952ffd640 10 client.4303.objecter ms_handle_connect 0x7f394406dc80 2024-03-29T11:37:05.628 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.628+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f395c050770 con 0x7f396413f9a0 2024-03-29T11:37:05.628 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.628+0000 7f3952ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:05.628 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.628+0000 7f3952ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:05.954 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.954+0000 7f3950ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:05.954 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.954+0000 7f3950ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:05.954 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.954+0000 7f3950ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:05.954 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.954+0000 7f3950ff9640 1 -- 172.21.15.145:0/3550567790 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3948005330 con 0x7f396413f9a0 2024-03-29T11:37:05.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.958+0000 7f3952ffd640 1 -- 172.21.15.145:0/3550567790 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f395c0526a0 con 0x7f396413f9a0 2024-03-29T11:37:05.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.958+0000 7f3952ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:05.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.958+0000 7f3952ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:05.958 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:37:05.958 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:05.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.961+0000 7f3950ff9640 10 librados: watch_flush enter 2024-03-29T11:37:05.964 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.963+0000 7f3950ff9640 10 librados: watch_flush exit 2024-03-29T11:37:05.964 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.963+0000 7f3950ff9640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-29T11:37:05.964 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.963+0000 7f3950ff9640 10 client.4303.objecter successfully canceled tick 2024-03-29T11:37:05.964 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.964+0000 7f3950ff9640 1 -- 172.21.15.145:0/3550567790 >> v1:172.21.15.145:6816/227372239 conn(0x7f394406dc80 legacy=0x7f3944070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:05.964 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.964+0000 7f3950ff9640 10 monclient: shutdown 2024-03-29T11:37:05.964 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.964+0000 7f3950ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:05.964 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.964+0000 7f3950ff9640 1 -- 172.21.15.145:0/3550567790 >> v1:172.21.15.203:6789/0 conn(0x7f396413f9a0 legacy=0x7f39641e6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:05.964 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.964+0000 7f3950ff9640 1 -- 172.21.15.145:0/3550567790 shutdown_connections 2024-03-29T11:37:05.964 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.964+0000 7f3950ff9640 1 -- 172.21.15.145:0/3550567790 >> 172.21.15.145:0/3550567790 conn(0x7f396406d230 msgr2=0x7f39640b0f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:05.965 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.964+0000 7f3950ff9640 1 -- 172.21.15.145:0/3550567790 shutdown_connections 2024-03-29T11:37:05.965 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.964+0000 7f3950ff9640 1 -- 172.21.15.145:0/3550567790 wait complete. 2024-03-29T11:37:05.965 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:05.964+0000 7f3950ff9640 1 librados: shutdown 2024-03-29T11:37:05.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:12.787 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:12.945 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.944+0000 7f7b7d2c8640 10 monclient: get_monmap_and_config 2024-03-29T11:37:12.945 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.944+0000 7f7b7d2c8640 10 monclient: build_initial_monmap 2024-03-29T11:37:12.945 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.944+0000 7f7b7d2c8640 10 monclient: monmap: 2024-03-29T11:37:12.945 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:12.945 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:12.945 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:37:12.946081+0000 2024-03-29T11:37:12.945 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:37:12.946081+0000 2024-03-29T11:37:12.946 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:12.946 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:12.946 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:12.946 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:37:12.946 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:37:12.946 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:12.946 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.946+0000 7f7b7d2c8640 1 Processor -- start 2024-03-29T11:37:12.946 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.946+0000 7f7b7d2c8640 1 -- start start 2024-03-29T11:37:12.946 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.946+0000 7f7b7d2c8640 10 monclient: init 2024-03-29T11:37:12.949 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.947+0000 7f7b7d2c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:12.949 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.947+0000 7f7b7d2c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:12.950 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.947+0000 7f7b7d2c8640 10 monclient(hunting): picked mon.noname-b con 0x7f7b780b0860 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:12.950 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.950+0000 7f7b7d2c8640 10 monclient(hunting): picked mon.noname-a con 0x7f7b7813ef30 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:12.951 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.950+0000 7f7b7d2c8640 10 monclient(hunting): picked mon.noname-c con 0x7f7b780af8e0 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:12.951 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.950+0000 7f7b7d2c8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b780b0c40 con 0x7f7b7813ef30 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.950+0000 7f7b7d2c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b780aed10 con 0x7f7b780af8e0 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.950+0000 7f7b7d2c8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b780afd50 con 0x7f7b780b0860 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.950+0000 7f7b7d2c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.950+0000 7f7b7d2c8640 10 monclient(hunting): authenticate will time out at 1022.258896s 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.951+0000 7f7b76ffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f7b780b0860 0x7f7b7813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:40151/0 (socket says 172.21.15.145:40151) 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.951+0000 7f7b76ffd640 1 -- 172.21.15.145:0/1082608127 learned_addr learned my addr 172.21.15.145:0/1082608127 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.951+0000 7f7b75ffb640 1 -- 172.21.15.145:0/1082608127 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737697006 0 0) 0x7f7b780afd50 con 0x7f7b780b0860 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.951+0000 7f7b75ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.951+0000 7f7b75ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.951+0000 7f7b75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.951+0000 7f7b75ffb640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:37:12.952 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.951+0000 7f7b75ffb640 1 -- 172.21.15.145:0/1082608127 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b60004630 con 0x7f7b780b0860 2024-03-29T11:37:12.956 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 1 -- 172.21.15.145:0/1082608127 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2122998261 0 0) 0x7f7b60004630 con 0x7f7b780b0860 2024-03-29T11:37:12.956 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 1 -- 172.21.15.145:0/1082608127 >> v1:172.21.15.203:6789/0 conn(0x7f7b780af8e0 legacy=0x7f7b78149a70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:37:12.956 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 1 -- 172.21.15.145:0/1082608127 >> v1:172.21.15.145:6789/0 conn(0x7f7b7813ef30 legacy=0x7f7b780af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 1 monclient: found mon.noname-b 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 1 -- 172.21.15.145:0/1082608127 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b780ab1b0 con 0x7f7b780b0860 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 1 -- 172.21.15.145:0/1082608127 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f7b6c004cb0 con 0x7f7b780b0860 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:12.957 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b75ffb640 10 monclient: dump: 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b7d2c8640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:37:12.958 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.954+0000 7f7b7d2c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.955+0000 7f7b75ffb640 1 -- 172.21.15.145:0/1082608127 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b6c004e80 con 0x7f7b780b0860 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.955+0000 7f7b75ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.955+0000 7f7b75ffb640 1 -- 172.21.15.145:0/1082608127 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f7b6c001430 con 0x7f7b780b0860 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.955+0000 7f7b75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.955+0000 7f7b75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.955+0000 7f7b75ffb640 10 monclient: dump: 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:12.959 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:12.960 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:12.960 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:12.960 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.955+0000 7f7b7d2c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:12.960 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.955+0000 7f7b7d2c8640 4 set_mon_vals no callback set 2024-03-29T11:37:12.960 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.956+0000 7f7b7d2c8640 10 monclient: shutdown 2024-03-29T11:37:12.960 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.956+0000 7f7b7d2c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:12.960 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.956+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/1082608127 >> v1:172.21.15.145:6790/0 conn(0x7f7b780b0860 legacy=0x7f7b7813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:12.960 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.956+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/1082608127 shutdown_connections 2024-03-29T11:37:12.960 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.956+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/1082608127 >> 172.21.15.145:0/1082608127 conn(0x7f7b7806b660 msgr2=0x7f7b7806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.957+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/1082608127 shutdown_connections 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.957+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/1082608127 wait complete. 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.958+0000 7f7b7d2c8640 10 monclient: build_initial_monmap 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.958+0000 7f7b7d2c8640 10 monclient: monmap: 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:12.962 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.959+0000 7f7b7d2c8640 1 librados: starting msgr at 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.959+0000 7f7b7d2c8640 1 librados: starting objecter 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.959+0000 7f7b7d2c8640 1 Processor -- start 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.959+0000 7f7b7d2c8640 1 -- start start 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.959+0000 7f7b7d2c8640 1 librados: setting wanted keys 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.959+0000 7f7b7d2c8640 1 librados: calling monclient init 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.959+0000 7f7b7d2c8640 10 monclient: init 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.960+0000 7f7b7d2c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:12.963 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.960+0000 7f7b7d2c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:12.965 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.960+0000 7f7b7d2c8640 10 monclient(hunting): picked mon.noname-c con 0x7f7b780af8e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:12.965 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.964+0000 7f7b7d2c8640 10 monclient(hunting): picked mon.noname-b con 0x7f7b780b0860 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:12.965 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.964+0000 7f7b7d2c8640 10 monclient(hunting): picked mon.noname-a con 0x7f7b7813ef30 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:12.965 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.964+0000 7f7b7d2c8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b7806ed30 con 0x7f7b7813ef30 2024-03-29T11:37:12.965 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.964+0000 7f7b7d2c8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b781d2f20 con 0x7f7b780b0860 2024-03-29T11:37:12.966 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.964+0000 7f7b7d2c8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b781379f0 con 0x7f7b780af8e0 2024-03-29T11:37:12.966 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.964+0000 7f7b7d2c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:12.966 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.964+0000 7f7b7d2c8640 10 monclient(hunting): authenticate will time out at 1022.272416s 2024-03-29T11:37:12.966 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.964+0000 7f7b76ffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f7b780af8e0 0x7f7b7806d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:40209/0 (socket says 172.21.15.145:40209) 2024-03-29T11:37:12.966 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.964+0000 7f7b76ffd640 1 -- 172.21.15.145:0/646688741 learned_addr learned my addr 172.21.15.145:0/646688741 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:12.967 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.966+0000 7f7b5f7fe640 10 client.?.objecter ms_handle_connect 0x7f7b780af8e0 2024-03-29T11:37:12.967 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.966+0000 7f7b5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:12.967 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.966+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136492084 0 0) 0x7f7b781379f0 con 0x7f7b780af8e0 2024-03-29T11:37:12.967 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.966+0000 7f7b5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:12.967 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.966+0000 7f7b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:12.967 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.966+0000 7f7b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:12.968 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.966+0000 7f7b5f7fe640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:37:12.968 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.966+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b64004eb0 con 0x7f7b780af8e0 2024-03-29T11:37:12.968 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.967+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4273793829 0 0) 0x7f7b64004eb0 con 0x7f7b780af8e0 2024-03-29T11:37:12.968 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.967+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b781379f0 con 0x7f7b780af8e0 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.967+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f7b6c004c30 con 0x7f7b780af8e0 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.967+0000 7f7b5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:12.969 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 10 client.?.objecter ms_handle_connect 0x7f7b780b0860 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996432149 0 0) 0x7f7b781d2f20 con 0x7f7b780b0860 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:37:12.970 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.968+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b64004eb0 con 0x7f7b780b0860 2024-03-29T11:37:12.971 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.969+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1784393461 0 0) 0x7f7b64004eb0 con 0x7f7b780b0860 2024-03-29T11:37:12.971 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.969+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b781d2f20 con 0x7f7b780b0860 2024-03-29T11:37:12.971 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.969+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f7b680016f0 con 0x7f7b780b0860 2024-03-29T11:37:12.971 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.969+0000 7f7b5f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:12.971 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.969+0000 7f7b5f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:12.971 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.969+0000 7f7b5f7fe640 10 monclient(hunting): dump: 2024-03-29T11:37:12.971 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:12.972 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:12.972 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.972 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.972 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:12.972 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:12.972 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:12.972 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:12.972 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:12.972 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:12.973 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.971+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2993380626 0 0) 0x7f7b781379f0 con 0x7f7b780af8e0 2024-03-29T11:37:12.973 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.971+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 >> v1:172.21.15.203:6789/0 conn(0x7f7b780b0860 legacy=0x7f7b781d3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.971+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 >> v1:172.21.15.145:6789/0 conn(0x7f7b7813ef30 legacy=0x7f7b78132680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 1 monclient: found mon.c 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b78135580 con 0x7f7b780af8e0 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b7d2c8640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b7d2c8640 10 monclient: _renew_subs 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b7d2c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:12.974 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b781d3b50 con 0x7f7b780af8e0 2024-03-29T11:37:12.975 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b7d2c8640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:12.975 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b7d2c8640 10 monclient: _renew_subs 2024-03-29T11:37:12.975 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b7d2c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:12.975 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b78138640 con 0x7f7b780af8e0 2024-03-29T11:37:12.975 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b7d2c8640 1 librados: init done 2024-03-29T11:37:12.975 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b6c0016f0 con 0x7f7b780af8e0 2024-03-29T11:37:12.975 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:12.975 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f7b6c001ce0 con 0x7f7b780af8e0 2024-03-29T11:37:12.975 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.972+0000 7f7b74ff9640 4 set_mon_vals no callback set 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.973+0000 7f7b5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.973+0000 7f7b5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.973+0000 7f7b5f7fe640 10 monclient: dump: 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:12.976 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:12.977 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:12.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.975+0000 7f7b7d2c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:12.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.975+0000 7f7b7d2c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:12.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.975+0000 7f7b7d2c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:12.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.975+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b58005080 con 0x7f7b780af8e0 2024-03-29T11:37:12.978 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.976+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f7b6c01ae60 con 0x7f7b780af8e0 2024-03-29T11:37:12.978 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.978+0000 7f7b767fc640 10 monclient: get_auth_request con 0x7f7b6406de40 auth_method 0 2024-03-29T11:37:12.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.979+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f7b6c084cd0 con 0x7f7b780af8e0 2024-03-29T11:37:12.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.979+0000 7f7b5f7fe640 10 client.4319.objecter ms_dispatch 0x7f7b78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:12.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.979+0000 7f7b5f7fe640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:12.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.979+0000 7f7b5f7fe640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:12.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.979+0000 7f7b5f7fe640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T11:37:12.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.980+0000 7f7b5f7fe640 10 client.4319.objecter ms_handle_connect 0x7f7b6406de40 2024-03-29T11:37:12.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.987+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b6c0505b0 con 0x7f7b780af8e0 2024-03-29T11:37:12.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.987+0000 7f7b5f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:12.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:12.987+0000 7f7b5f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:13.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.310+0000 7f7b7d2c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:13.312 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.310+0000 7f7b7d2c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:13.312 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.310+0000 7f7b7d2c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:13.312 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.310+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b58005330 con 0x7f7b780af8e0 2024-03-29T11:37:13.315 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.315+0000 7f7b5f7fe640 1 -- 172.21.15.145:0/646688741 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f7b6c0524e0 con 0x7f7b780af8e0 2024-03-29T11:37:13.315 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.315+0000 7f7b5f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:13.315 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.315+0000 7f7b5f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:13.315 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:37:13.316 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:13.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 10 librados: watch_flush enter 2024-03-29T11:37:13.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 10 librados: watch_flush exit 2024-03-29T11:37:13.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T11:37:13.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 10 client.4319.objecter successfully canceled tick 2024-03-29T11:37:13.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 >> v1:172.21.15.145:6816/227372239 conn(0x7f7b6406de40 legacy=0x7f7b64070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:13.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 10 monclient: shutdown 2024-03-29T11:37:13.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:13.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 >> v1:172.21.15.145:6790/0 conn(0x7f7b780af8e0 legacy=0x7f7b7806d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:13.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 shutdown_connections 2024-03-29T11:37:13.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.319+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 >> 172.21.15.145:0/646688741 conn(0x7f7b7806b660 msgr2=0x7f7b7813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:13.321 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.321+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 shutdown_connections 2024-03-29T11:37:13.321 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.322+0000 7f7b7d2c8640 1 -- 172.21.15.145:0/646688741 wait complete. 2024-03-29T11:37:13.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:13.322+0000 7f7b7d2c8640 1 librados: shutdown 2024-03-29T11:37:13.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:20.146 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:20.296 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.296+0000 7f8934fd7640 10 monclient: get_monmap_and_config 2024-03-29T11:37:20.296 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.296+0000 7f8934fd7640 10 monclient: build_initial_monmap 2024-03-29T11:37:20.296 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.296+0000 7f8934fd7640 10 monclient: monmap: 2024-03-29T11:37:20.296 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:37:20.297772+0000 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:37:20.297772+0000 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:20.297 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.297+0000 7f8934fd7640 1 Processor -- start 2024-03-29T11:37:20.301 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.297+0000 7f8934fd7640 1 -- start start 2024-03-29T11:37:20.301 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.297+0000 7f8934fd7640 10 monclient: init 2024-03-29T11:37:20.301 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:20.301 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 10 monclient(hunting): picked mon.noname-b con 0x7f893006c250 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 10 monclient(hunting): picked mon.noname-a con 0x7f8930072400 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 10 monclient(hunting): picked mon.noname-c con 0x7f893006c630 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893006d760 con 0x7f8930072400 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8930070500 con 0x7f893006c630 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300aff50 con 0x7f893006c250 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.298+0000 7f8934fd7640 10 monclient(hunting): authenticate will time out at 1029.606556s 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.301+0000 7f892ed9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f893006c630 0x7f8930151ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:35703/0 (socket says 172.21.15.145:35703) 2024-03-29T11:37:20.302 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.301+0000 7f892ed9d640 1 -- 172.21.15.145:0/617109809 learned_addr learned my addr 172.21.15.145:0/617109809 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.302+0000 7f892d59a640 1 -- 172.21.15.145:0/617109809 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877995657 0 0) 0x7f8930070500 con 0x7f893006c630 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.302+0000 7f892d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.302+0000 7f892d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.302+0000 7f892d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.302+0000 7f892d59a640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.302+0000 7f892d59a640 1 -- 172.21.15.145:0/617109809 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8918003520 con 0x7f893006c630 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.302+0000 7f892d59a640 1 -- 172.21.15.145:0/617109809 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378366024 0 0) 0x7f8918003520 con 0x7f893006c630 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.302+0000 7f892d59a640 1 -- 172.21.15.145:0/617109809 >> v1:172.21.15.145:6790/0 conn(0x7f893006c250 legacy=0x7f8930071cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 1 -- 172.21.15.145:0/617109809 >> v1:172.21.15.145:6789/0 conn(0x7f8930072400 legacy=0x7f89301465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:37:20.304 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 1 monclient: found mon.noname-c 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 1 -- 172.21.15.145:0/617109809 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300b00e0 con 0x7f893006c630 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 10 monclient: _finish_auth 0 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f8934fd7640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f8934fd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 1 -- 172.21.15.145:0/617109809 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8928001a10 con 0x7f893006c630 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 10 monclient: dump: 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:20.305 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f8934fd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 1 -- 172.21.15.145:0/617109809 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8928001ca0 con 0x7f893006c630 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f892d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f8934fd7640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:20.306 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.303+0000 7f8934fd7640 4 set_mon_vals no callback set 2024-03-29T11:37:20.307 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.304+0000 7f8934fd7640 10 monclient: shutdown 2024-03-29T11:37:20.307 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.304+0000 7f8934fd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:20.307 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.304+0000 7f8934fd7640 1 -- 172.21.15.145:0/617109809 >> v1:172.21.15.203:6789/0 conn(0x7f893006c630 legacy=0x7f8930151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.307 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.305+0000 7f8934fd7640 1 -- 172.21.15.145:0/617109809 shutdown_connections 2024-03-29T11:37:20.307 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.305+0000 7f8934fd7640 1 -- 172.21.15.145:0/617109809 >> 172.21.15.145:0/617109809 conn(0x7f8930067610 msgr2=0x7f89300b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:20.308 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.306+0000 7f8934fd7640 1 -- 172.21.15.145:0/617109809 shutdown_connections 2024-03-29T11:37:20.308 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.307+0000 7f8934fd7640 1 -- 172.21.15.145:0/617109809 wait complete. 2024-03-29T11:37:20.309 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.308+0000 7f8934fd7640 10 monclient: build_initial_monmap 2024-03-29T11:37:20.309 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.308+0000 7f8934fd7640 10 monclient: monmap: 2024-03-29T11:37:20.309 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:20.309 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:20.309 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.308+0000 7f8934fd7640 1 librados: starting msgr at 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.308+0000 7f8934fd7640 1 librados: starting objecter 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.308+0000 7f8934fd7640 1 Processor -- start 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.309+0000 7f8934fd7640 1 -- start start 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.309+0000 7f8934fd7640 1 librados: setting wanted keys 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.309+0000 7f8934fd7640 1 librados: calling monclient init 2024-03-29T11:37:20.310 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.309+0000 7f8934fd7640 10 monclient: init 2024-03-29T11:37:20.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:20.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:20.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 10 monclient(hunting): picked mon.noname-b con 0x7f893006c250 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:20.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 10 monclient(hunting): picked mon.noname-c con 0x7f893006c630 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:20.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 10 monclient(hunting): picked mon.noname-a con 0x7f8930072400 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:20.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300abca0 con 0x7f8930072400 2024-03-29T11:37:20.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89300aad70 con 0x7f893006c250 2024-03-29T11:37:20.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89301d9050 con 0x7f893006c630 2024-03-29T11:37:20.311 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:20.312 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.310+0000 7f8934fd7640 10 monclient(hunting): authenticate will time out at 1029.618386s 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f892ed9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f8930072400 0x7f89300b4000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:58542/0 (socket says 172.21.15.145:58542) 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f892ed9d640 1 -- 172.21.15.145:0/1128563832 learned_addr learned my addr 172.21.15.145:0/1128563832 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f891effd640 10 client.?.objecter ms_handle_connect 0x7f8930072400 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f891effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733765656 0 0) 0x7f89300abca0 con 0x7f8930072400 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f891effd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f891effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f891effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f891effd640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.314+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8910004eb0 con 0x7f8930072400 2024-03-29T11:37:20.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 507162997 0 0) 0x7f8910004eb0 con 0x7f8930072400 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89300abca0 con 0x7f8930072400 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8928004e60 con 0x7f8930072400 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 10 monclient(hunting): dump: 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:20.321 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2068782176 0 0) 0x7f89300abca0 con 0x7f8930072400 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 >> v1:172.21.15.145:6790/0 conn(0x7f893006c630 legacy=0x7f89300b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 >> v1:172.21.15.203:6789/0 conn(0x7f893006c250 legacy=0x7f89300aa580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 1 monclient: found mon.a 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300a9ab0 con 0x7f8930072400 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 10 monclient: _finish_auth 0 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f891effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f8934fd7640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:37:20.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f8934fd7640 10 monclient: _renew_subs 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f8934fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89300a9d00 con 0x7f8930072400 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f8934fd7640 10 client.4308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f8934fd7640 10 monclient: _renew_subs 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f8934fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89301e7290 con 0x7f8930072400 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.315+0000 7f8934fd7640 1 librados: init done 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.316+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89280052e0 con 0x7f8930072400 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.316+0000 7f891effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.316+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8928005730 con 0x7f8930072400 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.316+0000 7f891effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.316+0000 7f891effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.316+0000 7f891effd640 10 monclient: dump: 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:20.323 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.316+0000 7f892cd99640 4 set_mon_vals no callback set 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.318+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f8928001e90 con 0x7f8930072400 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f8928085150 con 0x7f8930072400 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f891effd640 10 client.4308.objecter ms_dispatch 0x7f8930000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f892e59c640 10 monclient: get_auth_request con 0x7f891006dd60 auth_method 0 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f891effd640 3 client.4308.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:20.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f891effd640 3 client.4308.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:20.325 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f8934fd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:20.325 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f891effd640 20 client.4308.objecter dump_active .. 0 homeless 2024-03-29T11:37:20.325 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f8934fd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:20.325 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f8934fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:37:20.325 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.319+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8914005080 con 0x7f8930072400 2024-03-29T11:37:20.325 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.320+0000 7f891effd640 10 client.4308.objecter ms_handle_connect 0x7f891006dd60 2024-03-29T11:37:20.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.330+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8928050a30 con 0x7f8930072400 2024-03-29T11:37:20.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.330+0000 7f891effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:20.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.330+0000 7f891effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:20.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.650+0000 7f8934fd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:20.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.650+0000 7f8934fd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:20.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.650+0000 7f8934fd7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:37:20.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.650+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8914005330 con 0x7f8930072400 2024-03-29T11:37:20.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.654+0000 7f891effd640 1 -- 172.21.15.145:0/1128563832 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f8928052960 con 0x7f8930072400 2024-03-29T11:37:20.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.654+0000 7f891effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:20.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.655+0000 7f891effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:20.655 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:37:20.655 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.658+0000 7f8934fd7640 10 librados: watch_flush enter 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.658+0000 7f8934fd7640 10 librados: watch_flush exit 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.658+0000 7f8934fd7640 20 client.4308.objecter shutdown clearing up homeless session... 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.658+0000 7f8934fd7640 10 client.4308.objecter successfully canceled tick 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.658+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 >> v1:172.21.15.145:6816/227372239 conn(0x7f891006dd60 legacy=0x7f8910070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.658+0000 7f8934fd7640 10 monclient: shutdown 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.658+0000 7f8934fd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.658+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 >> v1:172.21.15.145:6789/0 conn(0x7f8930072400 legacy=0x7f89300b4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.659+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 shutdown_connections 2024-03-29T11:37:20.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.659+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 >> 172.21.15.145:0/1128563832 conn(0x7f8930067610 msgr2=0x7f89300b2820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:20.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.659+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 shutdown_connections 2024-03-29T11:37:20.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.659+0000 7f8934fd7640 1 -- 172.21.15.145:0/1128563832 wait complete. 2024-03-29T11:37:20.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:20.659+0000 7f8934fd7640 1 librados: shutdown 2024-03-29T11:37:20.676 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:27.482 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.627+0000 7f4df2046640 10 monclient: get_monmap_and_config 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.627+0000 7f4df2046640 10 monclient: build_initial_monmap 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.628+0000 7f4df2046640 10 monclient: monmap: 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:37:27.629253+0000 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:37:27.629253+0000 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:27.628 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:37:27.629 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:37:27.629 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:27.629 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.629+0000 7f4df2046640 1 Processor -- start 2024-03-29T11:37:27.630 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.629+0000 7f4df2046640 1 -- start start 2024-03-29T11:37:27.630 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.629+0000 7f4df2046640 10 monclient: init 2024-03-29T11:37:27.630 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:27.630 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:27.630 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 10 monclient(hunting): picked mon.noname-c con 0x7f4de40389e0 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 10 monclient(hunting): picked mon.noname-a con 0x7f4de4038dc0 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 10 monclient(hunting): picked mon.noname-b con 0x7f4de4030660 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40c54c0 con 0x7f4de4038dc0 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de4036430 con 0x7f4de40389e0 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de4039840 con 0x7f4de4030660 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df2046640 10 monclient(hunting): authenticate will time out at 1036.938760s 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df1044640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f4de40389e0 0x7f4de4033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:45636/0 (socket says 172.21.15.145:45636) 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.630+0000 7f4df1044640 1 -- 172.21.15.145:0/1973171526 learned_addr learned my addr 172.21.15.145:0/1973171526 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:27.631 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672492491 0 0) 0x7f4de4039840 con 0x7f4de4030660 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd8003520 con 0x7f4de4030660 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1789491925 0 0) 0x7f4de4036430 con 0x7f4de40389e0 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:37:27.632 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.631+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd80048a0 con 0x7f4de40389e0 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 870721446 0 0) 0x7f4dd8003520 con 0x7f4de4030660 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 >> v1:172.21.15.203:6789/0 conn(0x7f4de40389e0 legacy=0x7f4de4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 >> v1:172.21.15.145:6789/0 conn(0x7f4de4038dc0 legacy=0x7f4de40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 1 monclient: found mon.noname-b 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de4035690 con 0x7f4de4030660 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 10 monclient: _finish_auth 0 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:27.634 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4dec04f730 con 0x7f4de4030660 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 10 monclient: dump: 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4df2046640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4df2046640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:27.635 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 1 -- 172.21.15.145:0/1973171526 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4dec075f10 con 0x7f4de4030660 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4de3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4df2046640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.632+0000 7f4df2046640 4 set_mon_vals no callback set 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.633+0000 7f4df2046640 10 monclient: shutdown 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.633+0000 7f4df2046640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.633+0000 7f4df2046640 1 -- 172.21.15.145:0/1973171526 >> v1:172.21.15.145:6790/0 conn(0x7f4de4030660 legacy=0x7f4de40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.633+0000 7f4df2046640 1 -- 172.21.15.145:0/1973171526 shutdown_connections 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.633+0000 7f4df2046640 1 -- 172.21.15.145:0/1973171526 >> 172.21.15.145:0/1973171526 conn(0x7f4de4005c00 msgr2=0x7f4de4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.633+0000 7f4df2046640 1 -- 172.21.15.145:0/1973171526 shutdown_connections 2024-03-29T11:37:27.636 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.633+0000 7f4df2046640 1 -- 172.21.15.145:0/1973171526 wait complete. 2024-03-29T11:37:27.644 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.634+0000 7f4df2046640 10 monclient: build_initial_monmap 2024-03-29T11:37:27.644 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.634+0000 7f4df2046640 10 monclient: monmap: 2024-03-29T11:37:27.644 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:27.644 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:27.644 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:37:27.644 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:37:27.644 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:27.644 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.635+0000 7f4df2046640 1 librados: starting msgr at 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.635+0000 7f4df2046640 1 librados: starting objecter 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.635+0000 7f4df2046640 1 Processor -- start 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.635+0000 7f4df2046640 1 -- start start 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.635+0000 7f4df2046640 1 librados: setting wanted keys 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.635+0000 7f4df2046640 1 librados: calling monclient init 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.635+0000 7f4df2046640 10 monclient: init 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 10 monclient(hunting): picked mon.noname-b con 0x7f4de4030660 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:27.645 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 10 monclient(hunting): picked mon.noname-c con 0x7f4de40389e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 10 monclient(hunting): picked mon.noname-a con 0x7f4de402f7f0 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de40a5af0 con 0x7f4de402f7f0 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de4160d50 con 0x7f4de4030660 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4de4160ee0 con 0x7f4de40389e0 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df2046640 10 monclient(hunting): authenticate will time out at 1036.944283s 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df1044640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f4de4030660 0x7f4de4160640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:45936/0 (socket says 172.21.15.145:45936) 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4df1044640 1 -- 172.21.15.145:0/1919184804 learned_addr learned my addr 172.21.15.145:0/1919184804 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4de1ffb640 10 client.?.objecter ms_handle_connect 0x7f4de4030660 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4de1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.636+0000 7f4de1ffb640 10 client.?.objecter ms_handle_connect 0x7f4de40389e0 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:27.646 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3567306818 0 0) 0x7f4de4160d50 con 0x7f4de4030660 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd0004eb0 con 0x7f4de4030660 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 client.?.objecter ms_handle_connect 0x7f4de402f7f0 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4184884393 0 0) 0x7f4dd0004eb0 con 0x7f4de4030660 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4de4160d50 con 0x7f4de4030660 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149753699 0 0) 0x7f4de4160ee0 con 0x7f4de40389e0 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.647 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dd0004eb0 con 0x7f4de40389e0 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825864523 0 0) 0x7f4de40a5af0 con 0x7f4de402f7f0 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4de4160ee0 con 0x7f4de402f7f0 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4ddc001a70 con 0x7f4de4030660 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:27.648 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.637+0000 7f4de1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 40355444 0 0) 0x7f4de4160d50 con 0x7f4de4030660 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 >> v1:172.21.15.145:6790/0 conn(0x7f4de40389e0 legacy=0x7f4de40c2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 >> v1:172.21.15.145:6789/0 conn(0x7f4de402f7f0 legacy=0x7f4de415cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 1 monclient: found mon.b 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:27.649 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de40c2ba0 con 0x7f4de4030660 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4de1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4df2046640 5 monclient: authenticate success, global_id 4315 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4df2046640 10 monclient: _renew_subs 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4df2046640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de40c2df0 con 0x7f4de4030660 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4df2046640 10 client.4315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4df2046640 10 monclient: _renew_subs 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4df2046640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de415d890 con 0x7f4de4030660 2024-03-29T11:37:27.650 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.638+0000 7f4df2046640 1 librados: init done 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.639+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ddc001bd0 con 0x7f4de4030660 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.639+0000 7f4de1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.639+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4ddc004c30 con 0x7f4de4030660 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.639+0000 7f4de1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.639+0000 7f4de1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.639+0000 7f4de1ffb640 10 monclient: dump: 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:27.651 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.639+0000 7f4de37fe640 4 set_mon_vals no callback set 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.641+0000 7f4df2046640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.641+0000 7f4df2046640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.641+0000 7f4df2046640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.641+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dcc005080 con 0x7f4de4030660 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.641+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f4ddc01ac00 con 0x7f4de4030660 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.642+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f4ddc084ae0 con 0x7f4de4030660 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.642+0000 7f4de1ffb640 10 client.4315.objecter ms_dispatch 0x7f4dec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.642+0000 7f4de1ffb640 3 client.4315.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.642+0000 7f4de1ffb640 3 client.4315.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.642+0000 7f4df0843640 10 monclient: get_auth_request con 0x7f4dd0076260 auth_method 0 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.642+0000 7f4de1ffb640 20 client.4315.objecter dump_active .. 0 homeless 2024-03-29T11:37:27.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.642+0000 7f4de1ffb640 10 client.4315.objecter ms_handle_connect 0x7f4dd0076260 2024-03-29T11:37:27.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.650+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ddc0503c0 con 0x7f4de4030660 2024-03-29T11:37:27.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.650+0000 7f4de1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:27.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:27.650+0000 7f4de1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:28.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.009+0000 7f4df2046640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:28.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.009+0000 7f4df2046640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:28.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.009+0000 7f4df2046640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:28.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.009+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dcc005330 con 0x7f4de4030660 2024-03-29T11:37:28.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.010+0000 7f4de1ffb640 1 -- 172.21.15.145:0/1919184804 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f4ddc0522f0 con 0x7f4de4030660 2024-03-29T11:37:28.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.011+0000 7f4de1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:28.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.011+0000 7f4de1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:28.011 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:37:28.011 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.014+0000 7f4df2046640 10 librados: watch_flush enter 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.014+0000 7f4df2046640 10 librados: watch_flush exit 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.014+0000 7f4df2046640 20 client.4315.objecter shutdown clearing up homeless session... 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.014+0000 7f4df2046640 10 client.4315.objecter successfully canceled tick 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.014+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 >> v1:172.21.15.145:6816/227372239 conn(0x7f4dd0076260 legacy=0x7f4dd0078720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.014+0000 7f4df2046640 10 monclient: shutdown 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.014+0000 7f4df2046640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.014+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 >> v1:172.21.15.203:6789/0 conn(0x7f4de4030660 legacy=0x7f4de4160640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.015+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 shutdown_connections 2024-03-29T11:37:28.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.015+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 >> 172.21.15.145:0/1919184804 conn(0x7f4de4005c00 msgr2=0x7f4de40c4470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:28.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.015+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 shutdown_connections 2024-03-29T11:37:28.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.015+0000 7f4df2046640 1 -- 172.21.15.145:0/1919184804 wait complete. 2024-03-29T11:37:28.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:28.015+0000 7f4df2046640 1 librados: shutdown 2024-03-29T11:37:28.030 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:34.836 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.976+0000 7f2dcc84e640 10 monclient: get_monmap_and_config 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.976+0000 7f2dcc84e640 10 monclient: build_initial_monmap 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.976+0000 7f2dcc84e640 10 monclient: monmap: 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:37:34.977228+0000 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:37:34.977228+0000 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:34.976 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:34.977 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:37:34.977 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:37:34.977 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:34.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.978+0000 7f2dcc84e640 1 Processor -- start 2024-03-29T11:37:34.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.978+0000 7f2dcc84e640 1 -- start start 2024-03-29T11:37:34.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.978+0000 7f2dcc84e640 10 monclient: init 2024-03-29T11:37:34.978 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc81457f0 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc813f650 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc80a9d70 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc80aaef0 con 0x7f2dc81457f0 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc8145bd0 con 0x7f2dc80a9d70 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc8145eb0 con 0x7f2dc813f650 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dcc84e640 10 monclient(hunting): authenticate will time out at 1044.286600s 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dc6d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f2dc80a9d70 0x7f2dc814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:48126/0 (socket says 172.21.15.145:48126) 2024-03-29T11:37:34.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.979+0000 7f2dc6d9d640 1 -- 172.21.15.145:0/1808733648 learned_addr learned my addr 172.21.15.145:0/1808733648 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:34.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.980+0000 7f2dc559a640 1 -- 172.21.15.145:0/1808733648 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493480534 0 0) 0x7f2dc8145bd0 con 0x7f2dc80a9d70 2024-03-29T11:37:34.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.980+0000 7f2dc559a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:34.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.980+0000 7f2dc559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:34.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.980+0000 7f2dc559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:34.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.980+0000 7f2dc559a640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:37:34.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.980+0000 7f2dc559a640 1 -- 172.21.15.145:0/1808733648 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0003520 con 0x7f2dc80a9d70 2024-03-29T11:37:34.981 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.980+0000 7f2dc559a640 1 -- 172.21.15.145:0/1808733648 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955108990 0 0) 0x7f2db0003520 con 0x7f2dc80a9d70 2024-03-29T11:37:34.981 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.980+0000 7f2dc559a640 1 -- 172.21.15.145:0/1808733648 >> v1:172.21.15.145:6790/0 conn(0x7f2dc813f650 legacy=0x7f2dc80b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:34.981 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.981+0000 7f2dc559a640 1 -- 172.21.15.145:0/1808733648 >> v1:172.21.15.145:6789/0 conn(0x7f2dc81457f0 legacy=0x7f2dc813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:34.982 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 10 monclient: _finish_hunting 0 2024-03-29T11:37:34.982 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 1 monclient: found mon.noname-c 2024-03-29T11:37:34.982 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:34.982 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 1 -- 172.21.15.145:0/1808733648 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc8146340 con 0x7f2dc80a9d70 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 10 monclient: _finish_auth 0 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 10 monclient: _check_auth_tickets 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 1 -- 172.21.15.145:0/1808733648 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2dbc001690 con 0x7f2dc80a9d70 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dc559a640 10 monclient: dump: 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:34.983 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:34.984 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:34.984 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:34.984 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:34.984 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:34.984 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dcc84e640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:37:34.984 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.983+0000 7f2dcc84e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:34.986 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.984+0000 7f2dc559a640 1 -- 172.21.15.145:0/1808733648 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dbc0018f0 con 0x7f2dc80a9d70 2024-03-29T11:37:34.986 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.984+0000 7f2dc559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.984+0000 7f2dc559a640 1 -- 172.21.15.145:0/1808733648 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2dbc001d40 con 0x7f2dc80a9d70 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.984+0000 7f2dc559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.984+0000 7f2dc559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.984+0000 7f2dc559a640 10 monclient: dump: 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:34.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.984+0000 7f2dcc84e640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.984+0000 7f2dcc84e640 4 set_mon_vals no callback set 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.984+0000 7f2dcc84e640 10 monclient: shutdown 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.985+0000 7f2dcc84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.985+0000 7f2dcc84e640 1 -- 172.21.15.145:0/1808733648 >> v1:172.21.15.203:6789/0 conn(0x7f2dc80a9d70 legacy=0x7f2dc814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.985+0000 7f2dcc84e640 1 -- 172.21.15.145:0/1808733648 shutdown_connections 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.985+0000 7f2dcc84e640 1 -- 172.21.15.145:0/1808733648 >> 172.21.15.145:0/1808733648 conn(0x7f2dc8066a00 msgr2=0x7f2dc8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.985+0000 7f2dcc84e640 1 -- 172.21.15.145:0/1808733648 shutdown_connections 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.985+0000 7f2dcc84e640 1 -- 172.21.15.145:0/1808733648 wait complete. 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.986+0000 7f2dcc84e640 10 monclient: build_initial_monmap 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.986+0000 7f2dcc84e640 10 monclient: monmap: 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:34.988 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.987+0000 7f2dcc84e640 1 librados: starting msgr at 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.987+0000 7f2dcc84e640 1 librados: starting objecter 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.987+0000 7f2dcc84e640 1 Processor -- start 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.987+0000 7f2dcc84e640 1 -- start start 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.987+0000 7f2dcc84e640 1 librados: setting wanted keys 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.987+0000 7f2dcc84e640 1 librados: calling monclient init 2024-03-29T11:37:34.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.987+0000 7f2dcc84e640 10 monclient: init 2024-03-29T11:37:34.990 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.988+0000 7f2dcc84e640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:34.990 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.988+0000 7f2dcc84e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:34.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.989+0000 7f2dcc84e640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc80a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:34.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.989+0000 7f2dcc84e640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc813f650 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:34.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.989+0000 7f2dcc84e640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc81457f0 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:34.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.989+0000 7f2dcc84e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc8135060 con 0x7f2dc81457f0 2024-03-29T11:37:34.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.989+0000 7f2dcc84e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc8135f10 con 0x7f2dc813f650 2024-03-29T11:37:34.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.990+0000 7f2dcc84e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc81361f0 con 0x7f2dc80a9d70 2024-03-29T11:37:34.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.990+0000 7f2dcc84e640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:34.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.990+0000 7f2dcc84e640 10 monclient(hunting): authenticate will time out at 1044.297046s 2024-03-29T11:37:34.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.992+0000 7f2dc5d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f2dc813f650 0x7f2dc8132d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:48680/0 (socket says 172.21.15.145:48680) 2024-03-29T11:37:34.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.992+0000 7f2dc5d9b640 1 -- 172.21.15.145:0/3007880682 learned_addr learned my addr 172.21.15.145:0/3007880682 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:34.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 client.?.objecter ms_handle_connect 0x7f2dc813f650 2024-03-29T11:37:34.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:34.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 client.?.objecter ms_handle_connect 0x7f2dc80a9d70 2024-03-29T11:37:34.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 935677250 0 0) 0x7f2dc81361f0 con 0x7f2dc80a9d70 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da8004eb0 con 0x7f2dc80a9d70 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060491475 0 0) 0x7f2dc8135f10 con 0x7f2dc813f650 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.993+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc81361f0 con 0x7f2dc813f650 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 client.?.objecter ms_handle_connect 0x7f2dc81457f0 2024-03-29T11:37:34.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1555006337 0 0) 0x7f2da8004eb0 con 0x7f2dc80a9d70 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dc8135f10 con 0x7f2dc80a9d70 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2dc0001a80 con 0x7f2dc80a9d70 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): dump: 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:34.997 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206048194 0 0) 0x7f2dc8135060 con 0x7f2dc81457f0 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da8004eb0 con 0x7f2dc81457f0 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 807219362 0 0) 0x7f2dc81361f0 con 0x7f2dc813f650 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dc8135060 con 0x7f2dc813f650 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2db8001920 con 0x7f2dc813f650 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:34.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.994+0000 7f2db6ffd640 10 monclient(hunting): dump: 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2438480384 0 0) 0x7f2dc8135f10 con 0x7f2dc80a9d70 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 >> v1:172.21.15.203:6789/0 conn(0x7f2dc813f650 legacy=0x7f2dc8132d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 >> v1:172.21.15.145:6789/0 conn(0x7f2dc81457f0 legacy=0x7f2dc8134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 1 monclient: found mon.c 2024-03-29T11:37:34.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc8135290 con 0x7f2dc80a9d70 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2db6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2dcc84e640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2dcc84e640 10 monclient: _renew_subs 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2dcc84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2dcc84e640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc8133e70 con 0x7f2dc80a9d70 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2dcc84e640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2dcc84e640 10 monclient: _renew_subs 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2dcc84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2dcc84e640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc81e3340 con 0x7f2dc80a9d70 2024-03-29T11:37:35.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.995+0000 7f2dcc84e640 1 librados: init done 2024-03-29T11:37:35.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.998+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc0002010 con 0x7f2dc80a9d70 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.999+0000 7f2db6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.999+0000 7f2db4ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:34.999+0000 7f2dc4d99640 4 set_mon_vals no callback set 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.000+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2dc0003b10 con 0x7f2dc80a9d70 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.000+0000 7f2db4ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.000+0000 7f2db4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.000+0000 7f2db4ff9640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dc80b24a0 con 0x7f2dc80a9d70 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.000+0000 7f2db6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.000+0000 7f2db6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.000+0000 7f2db6ffd640 10 monclient: dump: 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:35.003 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.000+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f2dc0019ae0 con 0x7f2dc80a9d70 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.001+0000 7f2dc5d9b640 10 monclient: get_auth_request con 0x7f2da806df50 auth_method 0 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.002+0000 7f2db6ffd640 10 client.4328.objecter ms_handle_connect 0x7f2da806df50 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.002+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f2dc0083ab0 con 0x7f2dc80a9d70 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.002+0000 7f2db6ffd640 10 client.4328.objecter ms_dispatch 0x7f2dc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.002+0000 7f2db6ffd640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.002+0000 7f2db6ffd640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:35.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.002+0000 7f2db6ffd640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:37:35.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.008+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dc004f2e0 con 0x7f2dc80a9d70 2024-03-29T11:37:35.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.008+0000 7f2db6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:35.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.008+0000 7f2db6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:35.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.334+0000 7f2db4ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:35.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.334+0000 7f2db4ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:35.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.334+0000 7f2db4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:35.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.334+0000 7f2db4ff9640 1 -- 172.21.15.145:0/3007880682 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dc8075f10 con 0x7f2dc80a9d70 2024-03-29T11:37:35.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.336+0000 7f2db6ffd640 1 -- 172.21.15.145:0/3007880682 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f2dc00513e0 con 0x7f2dc80a9d70 2024-03-29T11:37:35.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.336+0000 7f2db6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:35.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.336+0000 7f2db6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:35.335 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:37:35.336 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:35.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.339+0000 7f2db4ff9640 10 librados: watch_flush enter 2024-03-29T11:37:35.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.340+0000 7f2db4ff9640 10 librados: watch_flush exit 2024-03-29T11:37:35.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.340+0000 7f2db4ff9640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:37:35.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.340+0000 7f2db4ff9640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:37:35.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.340+0000 7f2db4ff9640 1 -- 172.21.15.145:0/3007880682 >> v1:172.21.15.145:6816/227372239 conn(0x7f2da806df50 legacy=0x7f2da8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:35.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.340+0000 7f2db4ff9640 10 monclient: shutdown 2024-03-29T11:37:35.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.340+0000 7f2db4ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:35.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.340+0000 7f2db4ff9640 1 -- 172.21.15.145:0/3007880682 >> v1:172.21.15.145:6790/0 conn(0x7f2dc80a9d70 legacy=0x7f2dc8136bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:35.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.341+0000 7f2db4ff9640 1 -- 172.21.15.145:0/3007880682 shutdown_connections 2024-03-29T11:37:35.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.341+0000 7f2db4ff9640 1 -- 172.21.15.145:0/3007880682 >> 172.21.15.145:0/3007880682 conn(0x7f2dc8066a00 msgr2=0x7f2dc80b60a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:35.342 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.343+0000 7f2db4ff9640 1 -- 172.21.15.145:0/3007880682 shutdown_connections 2024-03-29T11:37:35.343 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.344+0000 7f2db4ff9640 1 -- 172.21.15.145:0/3007880682 wait complete. 2024-03-29T11:37:35.343 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:35.344+0000 7f2db4ff9640 1 librados: shutdown 2024-03-29T11:37:35.360 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:42.166 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:42.315 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.314+0000 7fd82727b640 10 monclient: get_monmap_and_config 2024-03-29T11:37:42.315 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.314+0000 7fd82727b640 10 monclient: build_initial_monmap 2024-03-29T11:37:42.315 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.314+0000 7fd82727b640 10 monclient: monmap: 2024-03-29T11:37:42.315 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:42.315 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:42.315 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:37:42.315451+0000 2024-03-29T11:37:42.316 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:37:42.315451+0000 2024-03-29T11:37:42.316 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:42.316 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:42.316 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:42.316 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:37:42.316 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:37:42.316 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:42.316 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.315+0000 7fd82727b640 1 Processor -- start 2024-03-29T11:37:42.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.316+0000 7fd82727b640 1 -- start start 2024-03-29T11:37:42.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.316+0000 7fd82727b640 10 monclient: init 2024-03-29T11:37:42.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:42.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:37:42.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 10 monclient(hunting): picked mon.noname-a con 0x7fd8200b4100 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:42.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 10 monclient(hunting): picked mon.noname-c con 0x7fd82013ee90 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:42.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 10 monclient(hunting): picked mon.noname-b con 0x7fd8200af900 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:42.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200afd10 con 0x7fd8200b4100 2024-03-29T11:37:42.319 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200af6f0 con 0x7fd82013ee90 2024-03-29T11:37:42.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8200aac80 con 0x7fd8200af900 2024-03-29T11:37:42.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:42.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.317+0000 7fd82727b640 10 monclient(hunting): authenticate will time out at 1051.625258s 2024-03-29T11:37:42.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.319+0000 7fd825818640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fd8200af900 0x7fd82014be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:50251/0 (socket says 172.21.15.145:50251) 2024-03-29T11:37:42.320 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.319+0000 7fd825818640 1 -- 172.21.15.145:0/244327267 learned_addr learned my addr 172.21.15.145:0/244327267 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:42.321 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.319+0000 7fd817fff640 1 -- 172.21.15.145:0/244327267 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 80591170 0 0) 0x7fd8200aac80 con 0x7fd8200af900 2024-03-29T11:37:42.321 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.319+0000 7fd817fff640 10 monclient(hunting): _negotiate 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.319+0000 7fd817fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.319+0000 7fd817fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.319+0000 7fd817fff640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.319+0000 7fd817fff640 1 -- 172.21.15.145:0/244327267 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd80c003520 con 0x7fd8200af900 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 1 -- 172.21.15.145:0/244327267 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1808456488 0 0) 0x7fd80c003520 con 0x7fd8200af900 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 1 -- 172.21.15.145:0/244327267 >> v1:172.21.15.203:6789/0 conn(0x7fd82013ee90 legacy=0x7fd82013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 1 -- 172.21.15.145:0/244327267 >> v1:172.21.15.145:6789/0 conn(0x7fd8200b4100 legacy=0x7fd8200adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 10 monclient: _finish_hunting 0 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 1 monclient: found mon.noname-b 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 1 -- 172.21.15.145:0/244327267 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8200aaf60 con 0x7fd8200af900 2024-03-29T11:37:42.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 10 monclient: _finish_auth 0 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 10 monclient: _check_auth_tickets 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 1 -- 172.21.15.145:0/244327267 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd81c001690 con 0x7fd8200af900 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd817fff640 10 monclient: dump: 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:42.323 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd82727b640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.320+0000 7fd82727b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.321+0000 7fd817fff640 1 -- 172.21.15.145:0/244327267 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd81c001860 con 0x7fd8200af900 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.321+0000 7fd817fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.321+0000 7fd82727b640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.321+0000 7fd82727b640 4 set_mon_vals no callback set 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.321+0000 7fd817fff640 1 -- 172.21.15.145:0/244327267 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd81c001cb0 con 0x7fd8200af900 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.321+0000 7fd817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.321+0000 7fd817fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.321+0000 7fd817fff640 10 monclient: dump: 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:42.324 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:42.325 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:42.325 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:42.325 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:42.325 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:42.325 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:42.326 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.322+0000 7fd82727b640 10 monclient: shutdown 2024-03-29T11:37:42.326 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.322+0000 7fd82727b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:42.326 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.322+0000 7fd82727b640 1 -- 172.21.15.145:0/244327267 >> v1:172.21.15.145:6790/0 conn(0x7fd8200af900 legacy=0x7fd82014be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.326 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.322+0000 7fd82727b640 1 -- 172.21.15.145:0/244327267 shutdown_connections 2024-03-29T11:37:42.326 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.322+0000 7fd82727b640 1 -- 172.21.15.145:0/244327267 >> 172.21.15.145:0/244327267 conn(0x7fd8200b35e0 msgr2=0x7fd8200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:42.326 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.324+0000 7fd82727b640 1 -- 172.21.15.145:0/244327267 shutdown_connections 2024-03-29T11:37:42.326 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.324+0000 7fd82727b640 1 -- 172.21.15.145:0/244327267 wait complete. 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.325+0000 7fd82727b640 10 monclient: build_initial_monmap 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.325+0000 7fd82727b640 10 monclient: monmap: 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.325+0000 7fd82727b640 1 librados: starting msgr at 2024-03-29T11:37:42.327 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.325+0000 7fd82727b640 1 librados: starting objecter 2024-03-29T11:37:42.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.325+0000 7fd82727b640 1 Processor -- start 2024-03-29T11:37:42.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.325+0000 7fd82727b640 1 -- start start 2024-03-29T11:37:42.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.325+0000 7fd82727b640 1 librados: setting wanted keys 2024-03-29T11:37:42.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.325+0000 7fd82727b640 1 librados: calling monclient init 2024-03-29T11:37:42.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.325+0000 7fd82727b640 10 monclient: init 2024-03-29T11:37:42.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 10 monclient(hunting): picked mon.noname-a con 0x7fd8200af900 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 10 monclient(hunting): picked mon.noname-c con 0x7fd8200b4100 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 10 monclient(hunting): picked mon.noname-b con 0x7fd82013ee90 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd82013ded0 con 0x7fd8200af900 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd820129e20 con 0x7fd82013ee90 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8201e0500 con 0x7fd8200b4100 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.326+0000 7fd82727b640 10 monclient(hunting): authenticate will time out at 1051.634834s 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd824816640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fd8200b4100 0x7fd8201df6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:50369/0 (socket says 172.21.15.145:50369) 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd824816640 1 -- 172.21.15.145:0/3177626662 learned_addr learned my addr 172.21.15.145:0/3177626662 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 10 client.?.objecter ms_handle_connect 0x7fd8200b4100 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:42.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757038149 0 0) 0x7fd8201e0500 con 0x7fd8200b4100 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd804004eb0 con 0x7fd8200b4100 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 10 client.?.objecter ms_handle_connect 0x7fd82013ee90 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.329+0000 7fd815ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 287487352 0 0) 0x7fd804004eb0 con 0x7fd8200b4100 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8201e0500 con 0x7fd8200b4100 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd818004360 con 0x7fd8200b4100 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient(hunting): dump: 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:42.334 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 client.?.objecter ms_handle_connect 0x7fd8200af900 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959941654 0 0) 0x7fd820129e20 con 0x7fd82013ee90 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:42.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd804004eb0 con 0x7fd82013ee90 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1868072113 0 0) 0x7fd8201e0500 con 0x7fd8200b4100 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 >> v1:172.21.15.203:6789/0 conn(0x7fd82013ee90 legacy=0x7fd8201dfdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 >> v1:172.21.15.145:6789/0 conn(0x7fd8200af900 legacy=0x7fd82012a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 monclient: found mon.c 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd82012a090 con 0x7fd8200b4100 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient: _finish_auth 0 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd815ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd82727b640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd82727b640 10 monclient: _renew_subs 2024-03-29T11:37:42.336 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd82727b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.330+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd82012a370 con 0x7fd8200b4100 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd818001250 con 0x7fd8200b4100 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd815ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd8180015e0 con 0x7fd8200b4100 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd815ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd815ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd815ffb640 10 monclient: dump: 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:42.337 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:42.338 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:42.338 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd8177fe640 4 set_mon_vals no callback set 2024-03-29T11:37:42.338 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd82727b640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:42.338 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd82727b640 10 monclient: _renew_subs 2024-03-29T11:37:42.338 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd82727b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:42.338 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8201290a0 con 0x7fd8200b4100 2024-03-29T11:37:42.338 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.331+0000 7fd82727b640 1 librados: init done 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.332+0000 7fd82727b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.334+0000 7fd82727b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.334+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fd8180017c0 con 0x7fd8200b4100 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.334+0000 7fd82727b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.334+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8200abff0 con 0x7fd8200b4100 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.336+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fd818084bd0 con 0x7fd8200b4100 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.336+0000 7fd815ffb640 10 client.4334.objecter ms_dispatch 0x7fd820000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.336+0000 7fd815ffb640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.336+0000 7fd815ffb640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.336+0000 7fd815ffb640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.337+0000 7fd825017640 10 monclient: get_auth_request con 0x7fd80406df10 auth_method 0 2024-03-29T11:37:42.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.337+0000 7fd815ffb640 10 client.4334.objecter ms_handle_connect 0x7fd80406df10 2024-03-29T11:37:42.343 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.343+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8180505f0 con 0x7fd8200b4100 2024-03-29T11:37:42.343 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.343+0000 7fd815ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:42.343 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.343+0000 7fd815ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:42.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.664+0000 7fd82727b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:42.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.664+0000 7fd82727b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:42.666 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.664+0000 7fd82727b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:42.666 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.664+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd820075f10 con 0x7fd8200b4100 2024-03-29T11:37:42.666 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.665+0000 7fd815ffb640 1 -- 172.21.15.145:0/3177626662 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fd818001440 con 0x7fd8200b4100 2024-03-29T11:37:42.669 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:37:42.669 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:42.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.665+0000 7fd815ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:42.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.665+0000 7fd815ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:42.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.669+0000 7fd82727b640 10 librados: watch_flush enter 2024-03-29T11:37:42.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.669+0000 7fd82727b640 10 librados: watch_flush exit 2024-03-29T11:37:42.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.669+0000 7fd82727b640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T11:37:42.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.669+0000 7fd82727b640 10 client.4334.objecter successfully canceled tick 2024-03-29T11:37:42.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.669+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 >> v1:172.21.15.145:6816/227372239 conn(0x7fd80406df10 legacy=0x7fd8040703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.669+0000 7fd82727b640 10 monclient: shutdown 2024-03-29T11:37:42.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.670+0000 7fd82727b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:42.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.670+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 >> v1:172.21.15.145:6790/0 conn(0x7fd8200b4100 legacy=0x7fd8201df6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:42.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.670+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 shutdown_connections 2024-03-29T11:37:42.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.670+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 >> 172.21.15.145:0/3177626662 conn(0x7fd8200b35e0 msgr2=0x7fd820131270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:42.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.671+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 shutdown_connections 2024-03-29T11:37:42.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.671+0000 7fd82727b640 1 -- 172.21.15.145:0/3177626662 wait complete. 2024-03-29T11:37:42.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:42.671+0000 7fd82727b640 1 librados: shutdown 2024-03-29T11:37:42.687 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:49.494 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:49.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.648+0000 7f9a3087d640 10 monclient: get_monmap_and_config 2024-03-29T11:37:49.648 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.648+0000 7f9a3087d640 10 monclient: build_initial_monmap 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.648+0000 7f9a3087d640 10 monclient: monmap: 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:37:49.649412+0000 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:37:49.649412+0000 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:37:49.649 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:49.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.649+0000 7f9a3087d640 1 Processor -- start 2024-03-29T11:37:49.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.649+0000 7f9a3087d640 1 -- start start 2024-03-29T11:37:49.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.649+0000 7f9a3087d640 10 monclient: init 2024-03-29T11:37:49.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.650+0000 7f9a3087d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:49.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.650+0000 7f9a3087d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:37:49.652 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.650+0000 7f9a3087d640 10 monclient(hunting): picked mon.noname-c con 0x7f9a2c0b0860 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:49.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.652+0000 7f9a3087d640 10 monclient(hunting): picked mon.noname-b con 0x7f9a2c13ef30 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:49.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.652+0000 7f9a3087d640 10 monclient(hunting): picked mon.noname-a con 0x7f9a2c0af8e0 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:49.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.652+0000 7f9a3087d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c0b0c40 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.653+0000 7f9a3087d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c0aed10 con 0x7f9a2c0b0860 2024-03-29T11:37:49.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.653+0000 7f9a3087d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c0afd50 con 0x7f9a2c13ef30 2024-03-29T11:37:49.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.653+0000 7f9a3087d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:49.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.653+0000 7f9a3087d640 10 monclient(hunting): authenticate will time out at 1058.961060s 2024-03-29T11:37:49.657 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.657+0000 7f9a2ad9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f9a2c0af8e0 0x7f9a2c149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:53653/0 (socket says 172.21.15.145:53653) 2024-03-29T11:37:49.657 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.657+0000 7f9a2ad9d640 1 -- 172.21.15.145:0/3748297334 learned_addr learned my addr 172.21.15.145:0/3748297334 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:49.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.658+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902361719 0 0) 0x7f9a2c0b0c40 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.658+0000 7f9a2959a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:49.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.658+0000 7f9a2959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:49.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.658+0000 7f9a2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:49.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.658+0000 7f9a2959a640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:37:49.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.659+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a18003520 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590518588 0 0) 0x7f9a2c0aed10 con 0x7f9a2c0b0860 2024-03-29T11:37:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient(hunting): _negotiate 2024-03-29T11:37:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:37:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a2c0b0c40 con 0x7f9a2c0b0860 2024-03-29T11:37:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 406402223 0 0) 0x7f9a18003520 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 >> v1:172.21.15.145:6790/0 conn(0x7f9a2c13ef30 legacy=0x7f9a2c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:37:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 >> v1:172.21.15.203:6789/0 conn(0x7f9a2c0b0860 legacy=0x7f9a2c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: _finish_hunting 0 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 monclient: found mon.noname-a 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c0ab1b0 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: _finish_auth 0 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: _check_auth_tickets 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9a24001a10 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: dump: 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:49.662 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a3087d640 5 monclient: authenticate success, global_id 4323 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a3087d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a24001c30 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 1 -- 172.21.15.145:0/3748297334 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9a24001fc0 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:49.663 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a2959a640 10 monclient: dump: 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a3087d640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:49.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.660+0000 7f9a3087d640 4 set_mon_vals no callback set 2024-03-29T11:37:49.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.661+0000 7f9a3087d640 10 monclient: shutdown 2024-03-29T11:37:49.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.661+0000 7f9a3087d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:49.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.661+0000 7f9a3087d640 1 -- 172.21.15.145:0/3748297334 >> v1:172.21.15.145:6789/0 conn(0x7f9a2c0af8e0 legacy=0x7f9a2c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:49.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.661+0000 7f9a3087d640 1 -- 172.21.15.145:0/3748297334 shutdown_connections 2024-03-29T11:37:49.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.662+0000 7f9a3087d640 1 -- 172.21.15.145:0/3748297334 >> 172.21.15.145:0/3748297334 conn(0x7f9a2c06b660 msgr2=0x7f9a2c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:49.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.664+0000 7f9a3087d640 1 -- 172.21.15.145:0/3748297334 shutdown_connections 2024-03-29T11:37:49.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.664+0000 7f9a3087d640 1 -- 172.21.15.145:0/3748297334 wait complete. 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.665+0000 7f9a3087d640 10 monclient: build_initial_monmap 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.665+0000 7f9a3087d640 10 monclient: monmap: 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:49.667 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.666+0000 7f9a3087d640 1 librados: starting msgr at 2024-03-29T11:37:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.666+0000 7f9a3087d640 1 librados: starting objecter 2024-03-29T11:37:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.666+0000 7f9a3087d640 1 Processor -- start 2024-03-29T11:37:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.666+0000 7f9a3087d640 1 -- start start 2024-03-29T11:37:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.666+0000 7f9a3087d640 1 librados: setting wanted keys 2024-03-29T11:37:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.666+0000 7f9a3087d640 1 librados: calling monclient init 2024-03-29T11:37:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.666+0000 7f9a3087d640 10 monclient: init 2024-03-29T11:37:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.667+0000 7f9a3087d640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.667+0000 7f9a3087d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:37:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.667+0000 7f9a3087d640 10 monclient(hunting): picked mon.noname-c con 0x7f9a2c0af8e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.670+0000 7f9a3087d640 10 monclient(hunting): picked mon.noname-b con 0x7f9a2c0b0860 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.670+0000 7f9a3087d640 10 monclient(hunting): picked mon.noname-a con 0x7f9a2c13ef30 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.670+0000 7f9a3087d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c06ef80 con 0x7f9a2c13ef30 2024-03-29T11:37:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.670+0000 7f9a3087d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c06de20 con 0x7f9a2c0b0860 2024-03-29T11:37:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.670+0000 7f9a3087d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a2c135580 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.670+0000 7f9a3087d640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.670+0000 7f9a3087d640 10 monclient(hunting): authenticate will time out at 1058.978339s 2024-03-29T11:37:49.674 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.674+0000 7f9a29d9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f9a2c0b0860 0x7f9a2c1d3560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:50523/0 (socket says 172.21.15.145:50523) 2024-03-29T11:37:49.674 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.674+0000 7f9a29d9b640 1 -- 172.21.15.145:0/3163325249 learned_addr learned my addr 172.21.15.145:0/3163325249 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.674+0000 7f9a22ffd640 10 client.?.objecter ms_handle_connect 0x7f9a2c0b0860 2024-03-29T11:37:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.674+0000 7f9a22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.674+0000 7f9a22ffd640 10 client.?.objecter ms_handle_connect 0x7f9a2c0af8e0 2024-03-29T11:37:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.674+0000 7f9a22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.674+0000 7f9a22ffd640 10 client.?.objecter ms_handle_connect 0x7f9a2c13ef30 2024-03-29T11:37:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.674+0000 7f9a22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712384573 0 0) 0x7f9a2c06de20 con 0x7f9a2c0b0860 2024-03-29T11:37:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c004eb0 con 0x7f9a2c0b0860 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767688797 0 0) 0x7f9a2c06ef80 con 0x7f9a2c13ef30 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): my global_id is 4326 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a2c06de20 con 0x7f9a2c13ef30 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998256856 0 0) 0x7f9a2c135580 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.675+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a2c06ef80 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1173917173 0 0) 0x7f9a2c06ef80 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a2c135580 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9a14001930 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 10 monclient(hunting): dump: 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:49.682 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1336072680 0 0) 0x7f9a2c135580 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 >> v1:172.21.15.203:6789/0 conn(0x7f9a2c0b0860 legacy=0x7f9a2c1d3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 >> v1:172.21.15.145:6789/0 conn(0x7f9a2c13ef30 legacy=0x7f9a2c1dd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 1 monclient: found mon.c 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c135710 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 10 monclient: _finish_auth 0 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:37:49.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:49.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a3087d640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:37:49.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a3087d640 10 monclient: _renew_subs 2024-03-29T11:37:49.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a3087d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:49.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c138410 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a3087d640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:49.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a3087d640 10 monclient: _renew_subs 2024-03-29T11:37:49.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a3087d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:49.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c137b50 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.676+0000 7f9a3087d640 1 librados: init done 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.679+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a14001d40 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.679+0000 7f9a22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.679+0000 7f9a28d99640 4 set_mon_vals no callback set 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.680+0000 7f9a3087d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.680+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9a140037d0 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.680+0000 7f9a22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.680+0000 7f9a22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.680+0000 7f9a22ffd640 10 monclient: dump: 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:49.685 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.680+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f9a14019860 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.681+0000 7f9a3087d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.681+0000 7f9a3087d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.681+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a10005080 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.682+0000 7f9a29d9b640 10 monclient: get_auth_request con 0x7f9a0c06deb0 auth_method 0 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.682+0000 7f9a22ffd640 10 client.4337.objecter ms_handle_connect 0x7f9a0c06deb0 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.682+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f9a140838d0 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.682+0000 7f9a22ffd640 10 client.4337.objecter ms_dispatch 0x7f9a2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.682+0000 7f9a22ffd640 3 client.4337.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:49.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.682+0000 7f9a22ffd640 3 client.4337.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:49.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.682+0000 7f9a22ffd640 20 client.4337.objecter dump_active .. 0 homeless 2024-03-29T11:37:49.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.690+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a1404f060 con 0x7f9a2c0af8e0 2024-03-29T11:37:49.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.690+0000 7f9a22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:49.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:49.690+0000 7f9a22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:50.012 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.012+0000 7f9a3087d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:50.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.012+0000 7f9a3087d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:50.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.012+0000 7f9a3087d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:37:50.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.012+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a10005330 con 0x7f9a2c0af8e0 2024-03-29T11:37:50.014 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.013+0000 7f9a22ffd640 1 -- 172.21.15.145:0/3163325249 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f9a14051160 con 0x7f9a2c0af8e0 2024-03-29T11:37:50.014 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.013+0000 7f9a22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:50.014 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.013+0000 7f9a22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:50.014 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:37:50.014 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:50.020 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.017+0000 7f9a3087d640 10 librados: watch_flush enter 2024-03-29T11:37:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.020+0000 7f9a3087d640 10 librados: watch_flush exit 2024-03-29T11:37:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.020+0000 7f9a3087d640 20 client.4337.objecter shutdown clearing up homeless session... 2024-03-29T11:37:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.020+0000 7f9a3087d640 10 client.4337.objecter successfully canceled tick 2024-03-29T11:37:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.020+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 >> v1:172.21.15.145:6816/227372239 conn(0x7f9a0c06deb0 legacy=0x7f9a0c070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.020+0000 7f9a3087d640 10 monclient: shutdown 2024-03-29T11:37:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.020+0000 7f9a3087d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.020+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 >> v1:172.21.15.145:6790/0 conn(0x7f9a2c0af8e0 legacy=0x7f9a2c1d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.020+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 shutdown_connections 2024-03-29T11:37:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.020+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 >> 172.21.15.145:0/3163325249 conn(0x7f9a2c06b660 msgr2=0x7f9a2c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:50.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.021+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 shutdown_connections 2024-03-29T11:37:50.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.021+0000 7f9a3087d640 1 -- 172.21.15.145:0/3163325249 wait complete. 2024-03-29T11:37:50.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:50.021+0000 7f9a3087d640 1 librados: shutdown 2024-03-29T11:37:50.038 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:37:56.845 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:37:56.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.988+0000 7f334adc8640 10 monclient: get_monmap_and_config 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.988+0000 7f334adc8640 10 monclient: build_initial_monmap 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.988+0000 7f334adc8640 10 monclient: monmap: 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:37:56.989426+0000 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:37:56.989426+0000 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:37:56.989 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:56.990 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.990+0000 7f334adc8640 1 Processor -- start 2024-03-29T11:37:56.990 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.990+0000 7f334adc8640 1 -- start start 2024-03-29T11:37:56.990 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.990+0000 7f334adc8640 10 monclient: init 2024-03-29T11:37:56.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.991+0000 7f334adc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:56.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.991+0000 7f334adc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:56.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.991+0000 7f334adc8640 10 monclient(hunting): picked mon.noname-c con 0x7f333c037220 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:56.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.992+0000 7f334adc8640 10 monclient(hunting): picked mon.noname-a con 0x7f333c0cf8a0 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:56.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.992+0000 7f334adc8640 10 monclient(hunting): picked mon.noname-b con 0x7f333c02b0f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:56.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.992+0000 7f334adc8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333c02c120 con 0x7f333c0cf8a0 2024-03-29T11:37:56.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.992+0000 7f334adc8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333c02c400 con 0x7f333c037220 2024-03-29T11:37:56.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.992+0000 7f334adc8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333c02abd0 con 0x7f333c02b0f0 2024-03-29T11:37:56.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.992+0000 7f334adc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:56.992 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.992+0000 7f334adc8640 10 monclient(hunting): authenticate will time out at 1066.300188s 2024-03-29T11:37:56.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.995+0000 7f3343fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f333c0cf8a0 0x7f333c0c8bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:49487/0 (socket says 172.21.15.145:49487) 2024-03-29T11:37:56.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.995+0000 7f3348b64640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f333c037220 0x7f333c0cf190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:51529/0 (socket says 172.21.15.145:51529) 2024-03-29T11:37:56.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.995+0000 7f3348b64640 1 -- 172.21.15.145:0/1096216619 learned_addr learned my addr 172.21.15.145:0/1096216619 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:56.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.995+0000 7f33437fe640 1 -- 172.21.15.145:0/1096216619 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475582630 0 0) 0x7f333c02c120 con 0x7f333c0cf8a0 2024-03-29T11:37:56.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.995+0000 7f33437fe640 10 monclient(hunting): _negotiate 2024-03-29T11:37:56.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.995+0000 7f33437fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:56.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.995+0000 7f33437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:56.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.995+0000 7f33437fe640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:37:56.996 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.996+0000 7f33437fe640 1 -- 172.21.15.145:0/1096216619 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3334003520 con 0x7f333c0cf8a0 2024-03-29T11:37:56.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.996+0000 7f33437fe640 1 -- 172.21.15.145:0/1096216619 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 538712391 0 0) 0x7f3334003520 con 0x7f333c0cf8a0 2024-03-29T11:37:56.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.996+0000 7f33437fe640 1 -- 172.21.15.145:0/1096216619 >> v1:172.21.15.145:6790/0 conn(0x7f333c02b0f0 legacy=0x7f333c0d7530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.998+0000 7f33437fe640 1 -- 172.21.15.145:0/1096216619 >> v1:172.21.15.203:6789/0 conn(0x7f333c037220 legacy=0x7f333c0cf190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.998+0000 7f33437fe640 10 monclient: _finish_hunting 0 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.998+0000 7f33437fe640 1 monclient: found mon.noname-a 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.998+0000 7f33437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.998+0000 7f33437fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.998+0000 7f33437fe640 1 -- 172.21.15.145:0/1096216619 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c0c92c0 con 0x7f333c0cf8a0 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.998+0000 7f33437fe640 10 monclient: _finish_auth 0 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.998+0000 7f33437fe640 10 monclient: _check_auth_tickets 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.998+0000 7f33437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.999+0000 7f33437fe640 1 -- 172.21.15.145:0/1096216619 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3330001690 con 0x7f333c0cf8a0 2024-03-29T11:37:57.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.999+0000 7f334adc8640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.999+0000 7f334adc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.999+0000 7f33437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.999+0000 7f33437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.999+0000 7f33437fe640 10 monclient: dump: 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:57.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:56.999+0000 7f334adc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:37:57.006 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.005+0000 7f33437fe640 1 -- 172.21.15.145:0/1096216619 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3330001920 con 0x7f333c0cf8a0 2024-03-29T11:37:57.006 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.005+0000 7f33437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.006+0000 7f33437fe640 1 -- 172.21.15.145:0/1096216619 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3330001cb0 con 0x7f333c0cf8a0 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.006+0000 7f334adc8640 10 monclient: get_monmap_and_config success 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.006+0000 7f334adc8640 4 set_mon_vals no callback set 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.006+0000 7f33437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.006+0000 7f33437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.006+0000 7f33437fe640 10 monclient: dump: 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:57.007 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:57.008 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:57.008 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.007+0000 7f334adc8640 10 monclient: shutdown 2024-03-29T11:37:57.008 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.007+0000 7f334adc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.007+0000 7f334adc8640 1 -- 172.21.15.145:0/1096216619 >> v1:172.21.15.145:6789/0 conn(0x7f333c0cf8a0 legacy=0x7f333c0c8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.007+0000 7f334adc8640 1 -- 172.21.15.145:0/1096216619 shutdown_connections 2024-03-29T11:37:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.007+0000 7f334adc8640 1 -- 172.21.15.145:0/1096216619 >> 172.21.15.145:0/1096216619 conn(0x7f333c03a4b0 msgr2=0x7f333c0367f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.007+0000 7f334adc8640 1 -- 172.21.15.145:0/1096216619 shutdown_connections 2024-03-29T11:37:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.007+0000 7f334adc8640 1 -- 172.21.15.145:0/1096216619 wait complete. 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.008+0000 7f334adc8640 10 monclient: build_initial_monmap 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.008+0000 7f334adc8640 10 monclient: monmap: 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:37:57.010 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:37:57.011 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:57.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.009+0000 7f334adc8640 1 librados: starting msgr at 2024-03-29T11:37:57.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.009+0000 7f334adc8640 1 librados: starting objecter 2024-03-29T11:37:57.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.009+0000 7f334adc8640 1 Processor -- start 2024-03-29T11:37:57.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.009+0000 7f334adc8640 1 -- start start 2024-03-29T11:37:57.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.009+0000 7f334adc8640 1 librados: setting wanted keys 2024-03-29T11:37:57.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.009+0000 7f334adc8640 1 librados: calling monclient init 2024-03-29T11:37:57.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.009+0000 7f334adc8640 10 monclient: init 2024-03-29T11:37:57.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.010+0000 7f334adc8640 10 monclient: _reopen_session rank -1 2024-03-29T11:37:57.012 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.010+0000 7f334adc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:37:57.014 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.013+0000 7f334adc8640 10 monclient(hunting): picked mon.noname-b con 0x7f333c02b0f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:37:57.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.014+0000 7f334adc8640 10 monclient(hunting): picked mon.noname-a con 0x7f333c037220 addr v1:172.21.15.145:6789/0 2024-03-29T11:37:57.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.014+0000 7f3348b64640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f333c02b0f0 0x7f333c02de60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:51633/0 (socket says 172.21.15.145:51633) 2024-03-29T11:37:57.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.014+0000 7f3348b64640 1 -- 172.21.15.145:0/1965019093 learned_addr learned my addr 172.21.15.145:0/1965019093 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:37:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.014+0000 7f334adc8640 10 monclient(hunting): picked mon.noname-c con 0x7f333c161960 addr v1:172.21.15.145:6790/0 2024-03-29T11:37:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.018+0000 7f334adc8640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333c02e570 con 0x7f333c037220 2024-03-29T11:37:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.017+0000 7f33417fa640 10 client.?.objecter ms_handle_connect 0x7f333c02b0f0 2024-03-29T11:37:57.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.018+0000 7f33417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:57.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.018+0000 7f334adc8640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333c160d30 con 0x7f333c02b0f0 2024-03-29T11:37:57.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.018+0000 7f334adc8640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333c02c930 con 0x7f333c161960 2024-03-29T11:37:57.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.018+0000 7f334adc8640 10 monclient(hunting): _renew_subs 2024-03-29T11:37:57.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.018+0000 7f334adc8640 10 monclient(hunting): authenticate will time out at 1066.326518s 2024-03-29T11:37:57.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.020+0000 7f33417fa640 10 client.?.objecter ms_handle_connect 0x7f333c161960 2024-03-29T11:37:57.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.020+0000 7f33417fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:37:57.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.020+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2663117841 0 0) 0x7f333c160d30 con 0x7f333c02b0f0 2024-03-29T11:37:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.020+0000 7f33417fa640 10 monclient(hunting): _negotiate 2024-03-29T11:37:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.020+0000 7f33417fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:37:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.020+0000 7f33417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:37:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.020+0000 7f33417fa640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:37:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.020+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3328004eb0 con 0x7f333c02b0f0 2024-03-29T11:37:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.021+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3471237659 0 0) 0x7f3328004eb0 con 0x7f333c02b0f0 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.021+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f333c160d30 con 0x7f333c02b0f0 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.021+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f334404fdd0 con 0x7f333c02b0f0 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.021+0000 7f33417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.021+0000 7f33417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.021+0000 7f33417fa640 10 monclient(hunting): dump: 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:57.023 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.021+0000 7f33417fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1872611490 0 0) 0x7f333c160d30 con 0x7f333c02b0f0 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 >> v1:172.21.15.145:6790/0 conn(0x7f333c161960 legacy=0x7f333c0c29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 >> v1:172.21.15.145:6789/0 conn(0x7f333c037220 legacy=0x7f333c161250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 10 monclient: _finish_hunting 0 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 1 monclient: found mon.b 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c0cabf0 con 0x7f333c02b0f0 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 10 monclient: _finish_auth 0 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 10 monclient: _check_auth_tickets 2024-03-29T11:37:57.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.022+0000 7f33417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.023+0000 7f334adc8640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.023+0000 7f334adc8640 10 monclient: _renew_subs 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.023+0000 7f334adc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.023+0000 7f334adc8640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333c0c30d0 con 0x7f333c02b0f0 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.023+0000 7f334adc8640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.023+0000 7f334adc8640 10 monclient: _renew_subs 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.023+0000 7f334adc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.023+0000 7f334adc8640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f333c02d340 con 0x7f333c02b0f0 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.023+0000 7f334adc8640 1 librados: init done 2024-03-29T11:37:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.024+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f334404f400 con 0x7f333c02b0f0 2024-03-29T11:37:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.025+0000 7f332effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.025+0000 7f33417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:37:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.025+0000 7f33427fc640 4 set_mon_vals no callback set 2024-03-29T11:37:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.026+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3344075f10 con 0x7f333c02b0f0 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.027+0000 7f33417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.027+0000 7f33417fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.027+0000 7f33417fa640 10 monclient: dump: 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:37:57.028 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:37:57.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.027+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f3344067bf0 con 0x7f333c02b0f0 2024-03-29T11:37:57.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.028+0000 7f332effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:57.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.028+0000 7f332effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:57.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.028+0000 7f332effd640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333c004cd0 con 0x7f333c02b0f0 2024-03-29T11:37:57.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.028+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f33440ff670 con 0x7f333c02b0f0 2024-03-29T11:37:57.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.028+0000 7f33417fa640 10 client.4333.objecter ms_dispatch 0x7f3344000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:37:57.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.029+0000 7f33417fa640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:37:57.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.029+0000 7f33417fa640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:37:57.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.029+0000 7f33417fa640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T11:37:57.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.032+0000 7f3343fff640 10 monclient: get_auth_request con 0x7f332806da20 auth_method 0 2024-03-29T11:37:57.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.032+0000 7f33417fa640 10 client.4333.objecter ms_handle_connect 0x7f332806da20 2024-03-29T11:37:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.038+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33440caf50 con 0x7f333c02b0f0 2024-03-29T11:37:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.038+0000 7f33417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:37:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.038+0000 7f33417fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:37:57.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.360+0000 7f332effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:57.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.360+0000 7f332effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:57.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.360+0000 7f332effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:37:57.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.360+0000 7f332effd640 1 -- 172.21.15.145:0/1965019093 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f333c0cad80 con 0x7f333c02b0f0 2024-03-29T11:37:57.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.363+0000 7f33417fa640 1 -- 172.21.15.145:0/1965019093 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f33440cce80 con 0x7f333c02b0f0 2024-03-29T11:37:57.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.363+0000 7f33417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:37:57.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.363+0000 7f33417fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:37:57.364 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:37:57.365 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:37:57.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.368+0000 7f332effd640 10 librados: watch_flush enter 2024-03-29T11:37:57.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.368+0000 7f332effd640 10 librados: watch_flush exit 2024-03-29T11:37:57.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.368+0000 7f332effd640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T11:37:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.368+0000 7f332effd640 10 client.4333.objecter successfully canceled tick 2024-03-29T11:37:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.368+0000 7f332effd640 1 -- 172.21.15.145:0/1965019093 >> v1:172.21.15.145:6816/227372239 conn(0x7f332806da20 legacy=0x7f332806fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.368+0000 7f332effd640 10 monclient: shutdown 2024-03-29T11:37:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.368+0000 7f332effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:37:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.368+0000 7f332effd640 1 -- 172.21.15.145:0/1965019093 >> v1:172.21.15.203:6789/0 conn(0x7f333c02b0f0 legacy=0x7f333c02de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:37:57.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.369+0000 7f332effd640 1 -- 172.21.15.145:0/1965019093 shutdown_connections 2024-03-29T11:37:57.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.369+0000 7f332effd640 1 -- 172.21.15.145:0/1965019093 >> 172.21.15.145:0/1965019093 conn(0x7f333c03a4b0 msgr2=0x7f333c0380b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:37:57.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.371+0000 7f332effd640 1 -- 172.21.15.145:0/1965019093 shutdown_connections 2024-03-29T11:37:57.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.371+0000 7f332effd640 1 -- 172.21.15.145:0/1965019093 wait complete. 2024-03-29T11:37:57.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:37:57.371+0000 7f332effd640 1 librados: shutdown 2024-03-29T11:37:57.391 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:04.198 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:04.352 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.348+0000 7fcf75501640 10 monclient: get_monmap_and_config 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.348+0000 7fcf75501640 10 monclient: build_initial_monmap 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.349+0000 7fcf75501640 10 monclient: monmap: 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:38:04.350259+0000 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:38:04.350259+0000 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:38:04.353 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:04.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.354+0000 7fcf75501640 1 Processor -- start 2024-03-29T11:38:04.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.355+0000 7fcf75501640 1 -- start start 2024-03-29T11:38:04.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.355+0000 7fcf75501640 10 monclient: init 2024-03-29T11:38:04.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.356+0000 7fcf75501640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:04.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.356+0000 7fcf75501640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.356+0000 7fcf75501640 10 monclient(hunting): picked mon.noname-b con 0x7fcf700a9750 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.359+0000 7fcf75501640 10 monclient(hunting): picked mon.noname-a con 0x7fcf700adca0 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.359+0000 7fcf75501640 10 monclient(hunting): picked mon.noname-c con 0x7fcf70146a80 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.359+0000 7fcf75501640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf700abdc0 con 0x7fcf700adca0 2024-03-29T11:38:04.360 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.359+0000 7fcf75501640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf700b0070 con 0x7fcf70146a80 2024-03-29T11:38:04.360 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.359+0000 7fcf75501640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf700af880 con 0x7fcf700a9750 2024-03-29T11:38:04.360 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.359+0000 7fcf75501640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:04.360 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.359+0000 7fcf75501640 10 monclient(hunting): authenticate will time out at 1073.667390s 2024-03-29T11:38:04.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6ffff640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fcf700a9750 0x7fcf70143710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:56833/0 (socket says 172.21.15.145:56833) 2024-03-29T11:38:04.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6ffff640 1 -- 172.21.15.145:0/1093701303 learned_addr learned my addr 172.21.15.145:0/1093701303 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415632352 0 0) 0x7fcf700abdc0 con 0x7fcf700adca0 2024-03-29T11:38:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:38:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf58003520 con 0x7fcf700adca0 2024-03-29T11:38:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298290116 0 0) 0x7fcf700af880 con 0x7fcf700a9750 2024-03-29T11:38:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.362+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf700abdc0 con 0x7fcf700a9750 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3635544923 0 0) 0x7fcf58003520 con 0x7fcf700adca0 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 >> v1:172.21.15.145:6790/0 conn(0x7fcf700a9750 legacy=0x7fcf70143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 >> v1:172.21.15.203:6789/0 conn(0x7fcf70146a80 legacy=0x7fcf70151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 1 monclient: found mon.noname-a 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf700ac030 con 0x7fcf700adca0 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: _finish_auth 0 2024-03-29T11:38:04.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fcf640015c0 con 0x7fcf700adca0 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: dump: 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:04.368 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf640017e0 con 0x7fcf700adca0 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 1 -- 172.21.15.145:0/1093701303 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fcf64001b70 con 0x7fcf700adca0 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.363+0000 7fcf6effd640 10 monclient: dump: 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:04.369 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:04.370 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:04.370 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:04.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.364+0000 7fcf75501640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:38:04.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.364+0000 7fcf75501640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:04.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.364+0000 7fcf75501640 4 set_mon_vals no callback set 2024-03-29T11:38:04.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.364+0000 7fcf75501640 10 monclient: shutdown 2024-03-29T11:38:04.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.364+0000 7fcf75501640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.364+0000 7fcf75501640 1 -- 172.21.15.145:0/1093701303 >> v1:172.21.15.145:6789/0 conn(0x7fcf700adca0 legacy=0x7fcf70146370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.365+0000 7fcf75501640 1 -- 172.21.15.145:0/1093701303 shutdown_connections 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.365+0000 7fcf75501640 1 -- 172.21.15.145:0/1093701303 >> 172.21.15.145:0/1093701303 conn(0x7fcf700673e0 msgr2=0x7fcf700b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.365+0000 7fcf75501640 1 -- 172.21.15.145:0/1093701303 shutdown_connections 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.365+0000 7fcf75501640 1 -- 172.21.15.145:0/1093701303 wait complete. 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.365+0000 7fcf75501640 10 monclient: build_initial_monmap 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.365+0000 7fcf75501640 10 monclient: monmap: 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:04.375 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.366+0000 7fcf75501640 1 librados: starting msgr at 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.366+0000 7fcf75501640 1 librados: starting objecter 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.366+0000 7fcf75501640 1 Processor -- start 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.366+0000 7fcf75501640 1 -- start start 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.366+0000 7fcf75501640 1 librados: setting wanted keys 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.366+0000 7fcf75501640 1 librados: calling monclient init 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.366+0000 7fcf75501640 10 monclient: init 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.367+0000 7fcf75501640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.367+0000 7fcf75501640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.367+0000 7fcf75501640 10 monclient(hunting): picked mon.noname-a con 0x7fcf700a9750 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.370+0000 7fcf75501640 10 monclient(hunting): picked mon.noname-b con 0x7fcf700adca0 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:04.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.370+0000 7fcf75501640 10 monclient(hunting): picked mon.noname-c con 0x7fcf70146a80 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.370+0000 7fcf75501640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf70072420 con 0x7fcf700a9750 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.370+0000 7fcf75501640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf700725b0 con 0x7fcf700adca0 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.370+0000 7fcf75501640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf70072740 con 0x7fcf70146a80 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.370+0000 7fcf75501640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.370+0000 7fcf75501640 10 monclient(hunting): authenticate will time out at 1073.678900s 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.371+0000 7fcf6f7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fcf700adca0 0x7fcf7013e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:53811/0 (socket says 172.21.15.145:53811) 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.371+0000 7fcf6f7fe640 1 -- 172.21.15.145:0/3994251189 learned_addr learned my addr 172.21.15.145:0/3994251189 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.371+0000 7fcf6cff9640 10 client.?.objecter ms_handle_connect 0x7fcf700adca0 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.371+0000 7fcf6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.371+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991848530 0 0) 0x7fcf700725b0 con 0x7fcf700adca0 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:38:04.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf5c004eb0 con 0x7fcf700adca0 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550561444 0 0) 0x7fcf5c004eb0 con 0x7fcf700adca0 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf700725b0 con 0x7fcf700adca0 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fcf640015c0 con 0x7fcf700adca0 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 10 monclient(hunting): dump: 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:04.378 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:04.379 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:04.379 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:04.379 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2953001340 0 0) 0x7fcf700725b0 con 0x7fcf700adca0 2024-03-29T11:38:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.372+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 >> v1:172.21.15.145:6790/0 conn(0x7fcf70146a80 legacy=0x7fcf701cf260 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:04.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 >> v1:172.21.15.145:6789/0 conn(0x7fcf700a9750 legacy=0x7fcf701dea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:04.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 1 monclient: found mon.b 2024-03-29T11:38:04.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:04.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:04.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf701df190 con 0x7fcf700adca0 2024-03-29T11:38:04.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:04.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:04.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 10 client.?.objecter ms_handle_connect 0x7fcf700a9750 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf75501640 5 monclient: authenticate success, global_id 4336 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf75501640 10 monclient: _renew_subs 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf75501640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf75501640 1 -- 172.21.15.145:0/3994251189 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf701df400 con 0x7fcf700adca0 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.375+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf640038a0 con 0x7fcf700adca0 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf75501640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf75501640 10 monclient: _renew_subs 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf75501640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf75501640 1 -- 172.21.15.145:0/3994251189 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7013e9f0 con 0x7fcf700adca0 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf75501640 1 librados: init done 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf6cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fcf64003cf0 con 0x7fcf700adca0 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf6cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf6cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.376+0000 7fcf6cff9640 10 monclient: dump: 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.378+0000 7fcf567fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.378+0000 7fcf567fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.378+0000 7fcf567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.378+0000 7fcf567fc640 1 -- 172.21.15.145:0/3994251189 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf7004f730 con 0x7fcf700adca0 2024-03-29T11:38:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.378+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fcf64003e50 con 0x7fcf700adca0 2024-03-29T11:38:04.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.379+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fcf64084c70 con 0x7fcf700adca0 2024-03-29T11:38:04.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.379+0000 7fcf6cff9640 10 client.4336.objecter ms_dispatch 0x7fcf70000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:04.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.379+0000 7fcf6cff9640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:04.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.379+0000 7fcf6cff9640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:04.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.379+0000 7fcf6cff9640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-29T11:38:04.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.379+0000 7fcf6ffff640 10 monclient: get_auth_request con 0x7fcf5c06db80 auth_method 0 2024-03-29T11:38:04.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.379+0000 7fcf6e7fc640 4 set_mon_vals no callback set 2024-03-29T11:38:04.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.381+0000 7fcf6cff9640 10 client.4336.objecter ms_handle_connect 0x7fcf5c06db80 2024-03-29T11:38:04.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.386+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf64050550 con 0x7fcf700adca0 2024-03-29T11:38:04.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.386+0000 7fcf6cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:04.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.386+0000 7fcf6cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:04.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.710+0000 7fcf567fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:04.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.710+0000 7fcf567fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:04.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.710+0000 7fcf567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:04.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.710+0000 7fcf567fc640 1 -- 172.21.15.145:0/3994251189 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf701cef30 con 0x7fcf700adca0 2024-03-29T11:38:04.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.714+0000 7fcf6cff9640 1 -- 172.21.15.145:0/3994251189 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fcf64052480 con 0x7fcf700adca0 2024-03-29T11:38:04.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.714+0000 7fcf6cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:04.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.714+0000 7fcf6cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:04.715 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:38:04.716 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:04.718 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.718+0000 7fcf567fc640 10 librados: watch_flush enter 2024-03-29T11:38:04.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.720+0000 7fcf567fc640 10 librados: watch_flush exit 2024-03-29T11:38:04.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.720+0000 7fcf567fc640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-29T11:38:04.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.720+0000 7fcf567fc640 10 client.4336.objecter successfully canceled tick 2024-03-29T11:38:04.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.720+0000 7fcf567fc640 1 -- 172.21.15.145:0/3994251189 >> v1:172.21.15.145:6816/227372239 conn(0x7fcf5c06db80 legacy=0x7fcf5c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.720+0000 7fcf567fc640 10 monclient: shutdown 2024-03-29T11:38:04.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.720+0000 7fcf567fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:04.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.720+0000 7fcf567fc640 1 -- 172.21.15.145:0/3994251189 >> v1:172.21.15.203:6789/0 conn(0x7fcf700adca0 legacy=0x7fcf7013e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:04.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.720+0000 7fcf567fc640 1 -- 172.21.15.145:0/3994251189 shutdown_connections 2024-03-29T11:38:04.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.720+0000 7fcf567fc640 1 -- 172.21.15.145:0/3994251189 >> 172.21.15.145:0/3994251189 conn(0x7fcf700673e0 msgr2=0x7fcf700aa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:04.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.721+0000 7fcf567fc640 1 -- 172.21.15.145:0/3994251189 shutdown_connections 2024-03-29T11:38:04.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.721+0000 7fcf567fc640 1 -- 172.21.15.145:0/3994251189 wait complete. 2024-03-29T11:38:04.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:04.721+0000 7fcf567fc640 1 librados: shutdown 2024-03-29T11:38:04.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:11.548 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:11.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.696+0000 7f32aca42640 10 monclient: get_monmap_and_config 2024-03-29T11:38:11.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.696+0000 7f32aca42640 10 monclient: build_initial_monmap 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.696+0000 7f32aca42640 10 monclient: monmap: 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:38:11.698153+0000 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:38:11.698153+0000 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:38:11.698 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.698+0000 7f32aca42640 1 Processor -- start 2024-03-29T11:38:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.698+0000 7f32aca42640 1 -- start start 2024-03-29T11:38:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.698+0000 7f32aca42640 10 monclient: init 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 10 monclient(hunting): picked mon.noname-b con 0x7f32a80b2790 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 10 monclient(hunting): picked mon.noname-a con 0x7f32a8143c80 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 10 monclient(hunting): picked mon.noname-c con 0x7f32a8146e30 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a8146290 con 0x7f32a8143c80 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80b3970 con 0x7f32a8146e30 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a806ee80 con 0x7f32a80b2790 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.699+0000 7f32aca42640 10 monclient(hunting): authenticate will time out at 1081.007799s 2024-03-29T11:38:11.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.704+0000 7f32a659c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f32a8143c80 0x7f32a8146720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:51897/0 (socket says 172.21.15.145:51897) 2024-03-29T11:38:11.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.704+0000 7f32a659c640 1 -- 172.21.15.145:0/2831682969 learned_addr learned my addr 172.21.15.145:0/2831682969 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:11.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.704+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 262938629 0 0) 0x7f32a8146290 con 0x7f32a8143c80 2024-03-29T11:38:11.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.704+0000 7f32a5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:38:11.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.704+0000 7f32a5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:11.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.704+0000 7f32a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:11.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.704+0000 7f32a5d9b640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:38:11.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.704+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3290003520 con 0x7f32a8143c80 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.707+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326234493 0 0) 0x7f32a806ee80 con 0x7f32a80b2790 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.707+0000 7f32a5d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.707+0000 7f32a5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.707+0000 7f32a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.707+0000 7f32a5d9b640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.707+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32a8146290 con 0x7f32a80b2790 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3772015480 0 0) 0x7f32a8146290 con 0x7f32a80b2790 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 >> v1:172.21.15.203:6789/0 conn(0x7f32a8146e30 legacy=0x7f32a8151bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 >> v1:172.21.15.145:6789/0 conn(0x7f32a8143c80 legacy=0x7f32a8146720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 1 monclient: found mon.noname-b 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:38:11.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a806f160 con 0x7f32a80b2790 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 10 monclient: _finish_auth 0 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32aca42640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32aca42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3298001670 con 0x7f32a80b2790 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 10 monclient: dump: 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:11.712 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32aca42640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3298001890 con 0x7f32a80b2790 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.708+0000 7f32a5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.709+0000 7f32aca42640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.709+0000 7f32aca42640 4 set_mon_vals no callback set 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.709+0000 7f32a5d9b640 1 -- 172.21.15.145:0/2831682969 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3298001c20 con 0x7f32a80b2790 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.709+0000 7f32a5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.709+0000 7f32a5d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.709+0000 7f32a5d9b640 10 monclient: dump: 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:11.713 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.709+0000 7f32aca42640 10 monclient: shutdown 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.710+0000 7f32aca42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:11.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.710+0000 7f32aca42640 1 -- 172.21.15.145:0/2831682969 >> v1:172.21.15.145:6790/0 conn(0x7f32a80b2790 legacy=0x7f32a8143570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.710+0000 7f32aca42640 1 -- 172.21.15.145:0/2831682969 shutdown_connections 2024-03-29T11:38:11.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.710+0000 7f32aca42640 1 -- 172.21.15.145:0/2831682969 >> 172.21.15.145:0/2831682969 conn(0x7f32a806d070 msgr2=0x7f32a8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:11.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.712+0000 7f32aca42640 1 -- 172.21.15.145:0/2831682969 shutdown_connections 2024-03-29T11:38:11.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.715+0000 7f32aca42640 1 -- 172.21.15.145:0/2831682969 wait complete. 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.716+0000 7f32aca42640 10 monclient: build_initial_monmap 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.716+0000 7f32aca42640 10 monclient: monmap: 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:11.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.717+0000 7f32aca42640 1 librados: starting msgr at 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.717+0000 7f32aca42640 1 librados: starting objecter 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.717+0000 7f32aca42640 1 Processor -- start 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.717+0000 7f32aca42640 1 -- start start 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.717+0000 7f32aca42640 1 librados: setting wanted keys 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.717+0000 7f32aca42640 1 librados: calling monclient init 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.717+0000 7f32aca42640 10 monclient: init 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 10 monclient(hunting): picked mon.noname-c con 0x7f32a80b2790 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 10 monclient(hunting): picked mon.noname-a con 0x7f32a8143c80 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 10 monclient(hunting): picked mon.noname-b con 0x7f32a8146e30 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80b6e00 con 0x7f32a8143c80 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80b7290 con 0x7f32a8146e30 2024-03-29T11:38:11.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a81e19d0 con 0x7f32a80b2790 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.718+0000 7f32aca42640 10 monclient(hunting): authenticate will time out at 1081.026357s 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.721+0000 7f32a659c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f32a8143c80 0x7f32a81e0bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:52049/0 (socket says 172.21.15.145:52049) 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.721+0000 7f32a659c640 1 -- 172.21.15.145:0/2315351199 learned_addr learned my addr 172.21.15.145:0/2315351199 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.722+0000 7f328f7fe640 10 client.?.objecter ms_handle_connect 0x7f32a8143c80 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.722+0000 7f328f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.722+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1680745945 0 0) 0x7f32a80b6e00 con 0x7f32a8143c80 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.722+0000 7f328f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.722+0000 7f328f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.722+0000 7f328f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.722+0000 7f328f7fe640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.722+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3294004eb0 con 0x7f32a8143c80 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.723+0000 7f328f7fe640 10 client.?.objecter ms_handle_connect 0x7f32a80b2790 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.723+0000 7f328f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:11.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.723+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 349246336 0 0) 0x7f3294004eb0 con 0x7f32a8143c80 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.723+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32a80b6e00 con 0x7f32a8143c80 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.723+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f32a0001c90 con 0x7f32a8143c80 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.723+0000 7f328f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.723+0000 7f328f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.723+0000 7f328f7fe640 10 monclient(hunting): dump: 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:11.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.723+0000 7f328f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:11.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.724+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732465943 0 0) 0x7f32a81e19d0 con 0x7f32a80b2790 2024-03-29T11:38:11.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.724+0000 7f328f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:11.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.724+0000 7f328f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:11.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.724+0000 7f328f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:11.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.724+0000 7f328f7fe640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:38:11.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.724+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3294004eb0 con 0x7f32a80b2790 2024-03-29T11:38:11.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.724+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1341150849 0 0) 0x7f32a80b6e00 con 0x7f32a8143c80 2024-03-29T11:38:11.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.724+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 >> v1:172.21.15.145:6790/0 conn(0x7f32a80b2790 legacy=0x7f32a81e3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:11.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.724+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 >> v1:172.21.15.203:6789/0 conn(0x7f32a8146e30 legacy=0x7f32a81e12c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f328f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f328f7fe640 1 monclient: found mon.a 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f328f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f328f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a813f450 con 0x7f32a8143c80 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f328f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f328f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f328f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f32aca42640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f32aca42640 10 monclient: _renew_subs 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f32aca42640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:11.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a813f6c0 con 0x7f32a8143c80 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f32aca42640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f32aca42640 10 monclient: _renew_subs 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f32aca42640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a813eb40 con 0x7f32a8143c80 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.725+0000 7f32aca42640 1 librados: init done 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.727+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a0004b60 con 0x7f32a8143c80 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.727+0000 7f328f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.727+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f32a0004f30 con 0x7f32a8143c80 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.727+0000 7f328f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.727+0000 7f328f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.727+0000 7f328f7fe640 10 monclient: dump: 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:11.733 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.727+0000 7f32a559a640 4 set_mon_vals no callback set 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.729+0000 7f32aca42640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.729+0000 7f32aca42640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.729+0000 7f32aca42640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:11.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.729+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3288005080 con 0x7f32a8143c80 2024-03-29T11:38:11.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.731+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f32a0005090 con 0x7f32a8143c80 2024-03-29T11:38:11.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.732+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f32a0084db0 con 0x7f32a8143c80 2024-03-29T11:38:11.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.732+0000 7f328f7fe640 10 client.4341.objecter ms_dispatch 0x7f32a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:11.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.732+0000 7f328f7fe640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:11.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.732+0000 7f328f7fe640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:11.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.733+0000 7f328f7fe640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-29T11:38:11.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.733+0000 7f32a6d9d640 10 monclient: get_auth_request con 0x7f329406db70 auth_method 0 2024-03-29T11:38:11.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.733+0000 7f328f7fe640 10 client.4341.objecter ms_handle_connect 0x7f329406db70 2024-03-29T11:38:11.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.738+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32a0050690 con 0x7f32a8143c80 2024-03-29T11:38:11.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.738+0000 7f328f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:11.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:11.738+0000 7f328f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:12.065 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.065+0000 7f32aca42640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:12.066 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.065+0000 7f32aca42640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:12.066 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.065+0000 7f32aca42640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:12.066 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.065+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3288005330 con 0x7f32a8143c80 2024-03-29T11:38:12.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.067+0000 7f328f7fe640 1 -- 172.21.15.145:0/2315351199 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f32a00525c0 con 0x7f32a8143c80 2024-03-29T11:38:12.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.067+0000 7f328f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:12.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.067+0000 7f328f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:12.069 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:38:12.069 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:12.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.071+0000 7f32aca42640 10 librados: watch_flush enter 2024-03-29T11:38:12.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.071+0000 7f32aca42640 10 librados: watch_flush exit 2024-03-29T11:38:12.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.071+0000 7f32aca42640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-29T11:38:12.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.071+0000 7f32aca42640 10 client.4341.objecter successfully canceled tick 2024-03-29T11:38:12.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.072+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 >> v1:172.21.15.145:6816/227372239 conn(0x7f329406db70 legacy=0x7f3294070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:12.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.072+0000 7f32aca42640 10 monclient: shutdown 2024-03-29T11:38:12.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.072+0000 7f32aca42640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:12.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.072+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 >> v1:172.21.15.145:6789/0 conn(0x7f32a8143c80 legacy=0x7f32a81e0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:12.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.072+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 shutdown_connections 2024-03-29T11:38:12.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.072+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 >> 172.21.15.145:0/2315351199 conn(0x7f32a806d070 msgr2=0x7f32a8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:12.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.072+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 shutdown_connections 2024-03-29T11:38:12.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.072+0000 7f32aca42640 1 -- 172.21.15.145:0/2315351199 wait complete. 2024-03-29T11:38:12.074 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:12.072+0000 7f32aca42640 1 librados: shutdown 2024-03-29T11:38:12.091 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:18.898 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.054+0000 7ff5bda4a640 10 monclient: get_monmap_and_config 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.054+0000 7ff5bda4a640 10 monclient: build_initial_monmap 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.054+0000 7ff5bda4a640 10 monclient: monmap: 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:38:19.054707+0000 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:38:19.054707+0000 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:19.054 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:19.055 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:38:19.055 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:38:19.055 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:19.055 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.055+0000 7ff5bda4a640 1 Processor -- start 2024-03-29T11:38:19.055 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.055+0000 7ff5bda4a640 1 -- start start 2024-03-29T11:38:19.055 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.055+0000 7ff5bda4a640 10 monclient: init 2024-03-29T11:38:19.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:19.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:19.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff5b80b1f60 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:19.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff5b8071eb0 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:19.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff5b8143d10 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b806ef80 con 0x7ff5b8143d10 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b806f260 con 0x7ff5b80b1f60 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80b40c0 con 0x7ff5b8071eb0 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.056+0000 7ff5bda4a640 10 monclient(hunting): authenticate will time out at 1088.363695s 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.057+0000 7ff5b6ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7ff5b80b1f60 0x7ff5b8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:46232/0 (socket says 172.21.15.145:46232) 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.057+0000 7ff5b6ffd640 1 -- 172.21.15.145:0/2476206129 learned_addr learned my addr 172.21.15.145:0/2476206129 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.057+0000 7ff5b5ffb640 1 -- 172.21.15.145:0/2476206129 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291298015 0 0) 0x7ff5b806f260 con 0x7ff5b80b1f60 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.057+0000 7ff5b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.057+0000 7ff5b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.057+0000 7ff5b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:19.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.057+0000 7ff5b5ffb640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:38:19.058 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.057+0000 7ff5b5ffb640 1 -- 172.21.15.145:0/2476206129 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a0003520 con 0x7ff5b80b1f60 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 1 -- 172.21.15.145:0/2476206129 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1092744332 0 0) 0x7ff5a0003520 con 0x7ff5b80b1f60 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 1 -- 172.21.15.145:0/2476206129 >> v1:172.21.15.145:6790/0 conn(0x7ff5b8071eb0 legacy=0x7ff5b8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 1 -- 172.21.15.145:0/2476206129 >> v1:172.21.15.145:6789/0 conn(0x7ff5b8143d10 legacy=0x7ff5b8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 1 monclient: found mon.noname-c 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 1 -- 172.21.15.145:0/2476206129 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8155d70 con 0x7ff5b80b1f60 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 1 -- 172.21.15.145:0/2476206129 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff5b00019f0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:19.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: dump: 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5bda4a640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5bda4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 1 -- 172.21.15.145:0/2476206129 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5b0001bd0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:19.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5bda4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 1 -- 172.21.15.145:0/2476206129 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff5b0002020 con 0x7ff5b80b1f60 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5bda4a640 4 set_mon_vals no callback set 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.058+0000 7ff5b5ffb640 10 monclient: dump: 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.059+0000 7ff5bda4a640 10 monclient: shutdown 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.059+0000 7ff5bda4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:19.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.059+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2476206129 >> v1:172.21.15.203:6789/0 conn(0x7ff5b80b1f60 legacy=0x7ff5b8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:19.066 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.059+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2476206129 shutdown_connections 2024-03-29T11:38:19.066 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.059+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2476206129 >> 172.21.15.145:0/2476206129 conn(0x7ff5b8066a00 msgr2=0x7ff5b8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.060+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2476206129 shutdown_connections 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.060+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2476206129 wait complete. 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.061+0000 7ff5bda4a640 10 monclient: build_initial_monmap 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.061+0000 7ff5bda4a640 10 monclient: monmap: 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.061+0000 7ff5bda4a640 1 librados: starting msgr at 2024-03-29T11:38:19.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.061+0000 7ff5bda4a640 1 librados: starting objecter 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.061+0000 7ff5bda4a640 1 Processor -- start 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.062+0000 7ff5bda4a640 1 -- start start 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.062+0000 7ff5bda4a640 1 librados: setting wanted keys 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.062+0000 7ff5bda4a640 1 librados: calling monclient init 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.062+0000 7ff5bda4a640 10 monclient: init 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.062+0000 7ff5bda4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.062+0000 7ff5bda4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.062+0000 7ff5bda4a640 10 monclient(hunting): picked mon.noname-b con 0x7ff5b80b1f60 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.062+0000 7ff5bda4a640 10 monclient(hunting): picked mon.noname-a con 0x7ff5b8143d10 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff5bda4a640 10 monclient(hunting): picked mon.noname-c con 0x7ff5b81483d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff5bda4a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b8140990 con 0x7ff5b8143d10 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff5bda4a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b8147eb0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff5bda4a640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b81499d0 con 0x7ff5b81483d0 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff5bda4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:19.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff5bda4a640 10 monclient(hunting): authenticate will time out at 1088.370234s 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff5b67fc640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff5b8143d10 0x7ff5b813fb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:52484/0 (socket says 172.21.15.145:52484) 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff5b67fc640 1 -- 172.21.15.145:0/2938418610 learned_addr learned my addr 172.21.15.145:0/2938418610 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff59ffff640 10 client.?.objecter ms_handle_connect 0x7ff5b80b1f60 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff59ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff59ffff640 10 client.?.objecter ms_handle_connect 0x7ff5b81483d0 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.063+0000 7ff59ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3686807992 0 0) 0x7ff5b81499d0 con 0x7ff5b81483d0 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a4004eb0 con 0x7ff5b81483d0 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1918311584 0 0) 0x7ff5b8147eb0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:19.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b81499d0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 client.?.objecter ms_handle_connect 0x7ff5b8143d10 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.064+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 43907134 0 0) 0x7ff5a4004eb0 con 0x7ff5b81483d0 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5b8147eb0 con 0x7ff5b81483d0 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff5a8001740 con 0x7ff5b81483d0 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): dump: 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.070 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2326925751 0 0) 0x7ff5b81499d0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff5a4004eb0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff5b0001ee0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): dump: 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.071 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777346820 0 0) 0x7ff5b8140990 con 0x7ff5b8143d10 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.065+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b81499d0 con 0x7ff5b8143d10 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2927003671 0 0) 0x7ff5a4004eb0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 >> v1:172.21.15.145:6790/0 conn(0x7ff5b81483d0 legacy=0x7ff5b8140280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 >> v1:172.21.15.145:6789/0 conn(0x7ff5b8143d10 legacy=0x7ff5b813fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:19.072 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 1 monclient: found mon.b 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b8151fd0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 10 monclient: _finish_auth 0 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5bda4a640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5bda4a640 10 monclient: _renew_subs 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5bda4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b81521c0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5bda4a640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5bda4a640 10 monclient: _renew_subs 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5bda4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b8071020 con 0x7ff5b80b1f60 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5bda4a640 1 librados: init done 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff5b0002040 con 0x7ff5b80b1f60 2024-03-29T11:38:19.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff5b00015b0 con 0x7ff5b80b1f60 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff59ffff640 10 monclient: dump: 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:19.074 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.066+0000 7ff5b57fa640 4 set_mon_vals no callback set 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.067+0000 7ff5bda4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.067+0000 7ff5bda4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.067+0000 7ff5bda4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.067+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff598005080 con 0x7ff5b80b1f60 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.069+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7ff5b001af80 con 0x7ff5b80b1f60 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.070+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7ff5b0084e30 con 0x7ff5b80b1f60 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.070+0000 7ff59ffff640 10 client.4342.objecter ms_dispatch 0x7ff5b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.070+0000 7ff59ffff640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.070+0000 7ff59ffff640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:19.076 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.070+0000 7ff59ffff640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-29T11:38:19.077 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.070+0000 7ff5b67fc640 10 monclient: get_auth_request con 0x7ff5a406df70 auth_method 0 2024-03-29T11:38:19.077 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.070+0000 7ff59ffff640 10 client.4342.objecter ms_handle_connect 0x7ff5a406df70 2024-03-29T11:38:19.077 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.076+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff5b0050540 con 0x7ff5b80b1f60 2024-03-29T11:38:19.077 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.076+0000 7ff59ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:19.077 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.076+0000 7ff59ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:19.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.459+0000 7ff5bda4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:19.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.459+0000 7ff5bda4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:19.460 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.459+0000 7ff5bda4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:19.460 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.459+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff598005330 con 0x7ff5b80b1f60 2024-03-29T11:38:19.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.461+0000 7ff59ffff640 1 -- 172.21.15.145:0/2938418610 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7ff5b0052640 con 0x7ff5b80b1f60 2024-03-29T11:38:19.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.461+0000 7ff59ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:19.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.461+0000 7ff59ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:19.461 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:38:19.461 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:19.464 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 10 librados: watch_flush enter 2024-03-29T11:38:19.464 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 10 librados: watch_flush exit 2024-03-29T11:38:19.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-29T11:38:19.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 10 client.4342.objecter successfully canceled tick 2024-03-29T11:38:19.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 >> v1:172.21.15.145:6816/227372239 conn(0x7ff5a406df70 legacy=0x7ff5a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:19.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 10 monclient: shutdown 2024-03-29T11:38:19.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:19.465 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 >> v1:172.21.15.203:6789/0 conn(0x7ff5b80b1f60 legacy=0x7ff5b81492c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:19.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 shutdown_connections 2024-03-29T11:38:19.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 >> 172.21.15.145:0/2938418610 conn(0x7ff5b8066a00 msgr2=0x7ff5b80b7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:19.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.465+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 shutdown_connections 2024-03-29T11:38:19.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.466+0000 7ff5bda4a640 1 -- 172.21.15.145:0/2938418610 wait complete. 2024-03-29T11:38:19.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:19.466+0000 7ff5bda4a640 1 librados: shutdown 2024-03-29T11:38:19.482 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:26.289 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:26.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.448+0000 7f229cdc6640 10 monclient: get_monmap_and_config 2024-03-29T11:38:26.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.448+0000 7f229cdc6640 10 monclient: build_initial_monmap 2024-03-29T11:38:26.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.448+0000 7f229cdc6640 10 monclient: monmap: 2024-03-29T11:38:26.448 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:26.448 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:26.449 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:38:26.448753+0000 2024-03-29T11:38:26.449 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:38:26.448753+0000 2024-03-29T11:38:26.449 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:26.449 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:26.449 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:26.449 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:38:26.449 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:38:26.449 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:26.449 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.449+0000 7f229cdc6640 1 Processor -- start 2024-03-29T11:38:26.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.449+0000 7f229cdc6640 1 -- start start 2024-03-29T11:38:26.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.449+0000 7f229cdc6640 10 monclient: init 2024-03-29T11:38:26.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:26.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:26.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f22981422b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f2298142690 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f22980b17d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2298142c80 con 0x7f2298142690 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22980aae40 con 0x7f22981422b0 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f229813fbd0 con 0x7f22980b17d0 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.450+0000 7f229cdc6640 10 monclient(hunting): authenticate will time out at 1095.757964s 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f2296d9d640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f22980b17d0 0x7f22980b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:49184/0 (socket says 172.21.15.145:49184) 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229659c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f22981422b0 0x7f229813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:57398/0 (socket says 172.21.15.145:57398) 2024-03-29T11:38:26.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f2296d9d640 1 -- 172.21.15.145:0/3968127128 learned_addr learned my addr 172.21.15.145:0/3968127128 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169912322 0 0) 0x7f229813fbd0 con 0x7f22980b17d0 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2280003520 con 0x7f22980b17d0 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545356215 0 0) 0x7f22980aae40 con 0x7f22981422b0 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:38:26.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.451+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f229813fbd0 con 0x7f22981422b0 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1794246507 0 0) 0x7f2280003520 con 0x7f22980b17d0 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 >> v1:172.21.15.203:6789/0 conn(0x7f22981422b0 legacy=0x7f229813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 >> v1:172.21.15.145:6789/0 conn(0x7f2298142690 legacy=0x7f22980b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 1 monclient: found mon.noname-b 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2298151940 con 0x7f22980b17d0 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 10 monclient: _finish_auth 0 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229cdc6640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229cdc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:26.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2290001940 con 0x7f22980b17d0 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229559a640 10 monclient: dump: 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:26.455 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.452+0000 7f229cdc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:26.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2290001b60 con 0x7f22980b17d0 2024-03-29T11:38:26.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229cdc6640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229cdc6640 4 set_mon_vals no callback set 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229559a640 1 -- 172.21.15.145:0/3968127128 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2290001e70 con 0x7f22980b17d0 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229cdc6640 10 monclient: shutdown 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229559a640 10 monclient: dump: 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.458 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229cdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.453+0000 7f229cdc6640 1 -- 172.21.15.145:0/3968127128 >> v1:172.21.15.145:6790/0 conn(0x7f22980b17d0 legacy=0x7f22980b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.454+0000 7f229cdc6640 1 -- 172.21.15.145:0/3968127128 shutdown_connections 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.454+0000 7f229cdc6640 1 -- 172.21.15.145:0/3968127128 >> 172.21.15.145:0/3968127128 conn(0x7f2298066a00 msgr2=0x7f2298066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.454+0000 7f229cdc6640 1 -- 172.21.15.145:0/3968127128 shutdown_connections 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.454+0000 7f229cdc6640 1 -- 172.21.15.145:0/3968127128 wait complete. 2024-03-29T11:38:26.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.455+0000 7f229cdc6640 10 monclient: build_initial_monmap 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.455+0000 7f229cdc6640 10 monclient: monmap: 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.456+0000 7f229cdc6640 1 librados: starting msgr at 2024-03-29T11:38:26.460 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.456+0000 7f229cdc6640 1 librados: starting objecter 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.456+0000 7f229cdc6640 1 Processor -- start 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.456+0000 7f229cdc6640 1 -- start start 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.456+0000 7f229cdc6640 1 librados: setting wanted keys 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.456+0000 7f229cdc6640 1 librados: calling monclient init 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.456+0000 7f229cdc6640 10 monclient: init 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 10 monclient(hunting): picked mon.noname-c con 0x7f22980b17d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 10 monclient(hunting): picked mon.noname-a con 0x7f22981422b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 10 monclient(hunting): picked mon.noname-b con 0x7f2298142690 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2298137250 con 0x7f22981422b0 2024-03-29T11:38:26.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22980b6f00 con 0x7f2298142690 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22980b71e0 con 0x7f22980b17d0 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229cdc6640 10 monclient(hunting): authenticate will time out at 1095.764359s 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229659c640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f22980b17d0 0x7f22980b5c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:49310/0 (socket says 172.21.15.145:49310) 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f229659c640 1 -- 172.21.15.145:0/3713454545 learned_addr learned my addr 172.21.15.145:0/3713454545 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 client.?.objecter ms_handle_connect 0x7f22980b17d0 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 client.?.objecter ms_handle_connect 0x7f22981422b0 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 client.?.objecter ms_handle_connect 0x7f2298142690 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:26.462 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665212549 0 0) 0x7f22980b71e0 con 0x7f22980b17d0 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.457+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2278004eb0 con 0x7f22980b17d0 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099573101 0 0) 0x7f2298142c80 con 0x7f22981422b0 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22980b71e0 con 0x7f22981422b0 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019473982 0 0) 0x7f22980b6f00 con 0x7f2298142690 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:26.463 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:26.464 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:26.464 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:38:26.464 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.458+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2298142c80 con 0x7f2298142690 2024-03-29T11:38:26.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 523992512 0 0) 0x7f2278004eb0 con 0x7f22980b17d0 2024-03-29T11:38:26.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22980b6f00 con 0x7f22980b17d0 2024-03-29T11:38:26.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f22880019f0 con 0x7f22980b17d0 2024-03-29T11:38:26.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:26.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:26.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:26.466 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2029644298 0 0) 0x7f2298142c80 con 0x7f2298142690 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2278004eb0 con 0x7f2298142690 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2290001800 con 0x7f2298142690 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:26.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient(hunting): dump: 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4168212446 0 0) 0x7f2278004eb0 con 0x7f2298142690 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 >> v1:172.21.15.145:6790/0 conn(0x7f22980b17d0 legacy=0x7f22980b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 >> v1:172.21.15.145:6789/0 conn(0x7f22981422b0 legacy=0x7f22981362e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 monclient: found mon.b 2024-03-29T11:38:26.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22980b7600 con 0x7f2298142690 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f2286ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f229cdc6640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f229cdc6640 10 monclient: _renew_subs 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f229cdc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.459+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22981510c0 con 0x7f2298142690 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.460+0000 7f229cdc6640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.460+0000 7f229cdc6640 10 monclient: _renew_subs 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.460+0000 7f229cdc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.460+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22981413e0 con 0x7f2298142690 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.460+0000 7f229cdc6640 1 librados: init done 2024-03-29T11:38:26.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.461+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2290001960 con 0x7f2298142690 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.461+0000 7f2286ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.461+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2290004fb0 con 0x7f2298142690 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.461+0000 7f2286ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.461+0000 7f2286ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.461+0000 7f2286ffd640 10 monclient: dump: 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:26.470 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.461+0000 7f2287fff640 4 set_mon_vals no callback set 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.461+0000 7f229cdc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.462+0000 7f229cdc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.462+0000 7f229cdc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.462+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f227c005080 con 0x7f2298142690 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.463+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f229001af80 con 0x7f2298142690 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.464+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f2290084d60 con 0x7f2298142690 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.464+0000 7f2286ffd640 10 client.4348.objecter ms_dispatch 0x7f2298000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.464+0000 7f2286ffd640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.464+0000 7f2286ffd640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.464+0000 7f2286ffd640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.464+0000 7f229659c640 10 monclient: get_auth_request con 0x7f227806df50 auth_method 0 2024-03-29T11:38:26.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.464+0000 7f2286ffd640 10 client.4348.objecter ms_handle_connect 0x7f227806df50 2024-03-29T11:38:26.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.470+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22900506a0 con 0x7f2298142690 2024-03-29T11:38:26.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.470+0000 7f2286ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:26.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.470+0000 7f2286ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:26.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.854+0000 7f229cdc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:26.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.854+0000 7f229cdc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:26.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.854+0000 7f229cdc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:26.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.854+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f227c005330 con 0x7f2298142690 2024-03-29T11:38:26.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.856+0000 7f2286ffd640 1 -- 172.21.15.145:0/3713454545 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f22900506a0 con 0x7f2298142690 2024-03-29T11:38:26.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.856+0000 7f2286ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:26.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.856+0000 7f2286ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:26.856 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:38:26.856 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:26.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 10 librados: watch_flush enter 2024-03-29T11:38:26.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 10 librados: watch_flush exit 2024-03-29T11:38:26.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:38:26.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:38:26.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 >> v1:172.21.15.145:6816/227372239 conn(0x7f227806df50 legacy=0x7f2278070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 10 monclient: shutdown 2024-03-29T11:38:26.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:26.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 >> v1:172.21.15.203:6789/0 conn(0x7f2298142690 legacy=0x7f22981fffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:26.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 shutdown_connections 2024-03-29T11:38:26.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 >> 172.21.15.145:0/3713454545 conn(0x7f2298066a00 msgr2=0x7f229813e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:26.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 shutdown_connections 2024-03-29T11:38:26.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 1 -- 172.21.15.145:0/3713454545 wait complete. 2024-03-29T11:38:26.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:26.860+0000 7f229cdc6640 1 librados: shutdown 2024-03-29T11:38:26.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:33.682 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:33.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.829+0000 7f72a3323640 10 monclient: get_monmap_and_config 2024-03-29T11:38:33.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.829+0000 7f72a3323640 10 monclient: build_initial_monmap 2024-03-29T11:38:33.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.829+0000 7f72a3323640 10 monclient: monmap: 2024-03-29T11:38:33.831 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:33.832 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:33.832 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:38:33.830991+0000 2024-03-29T11:38:33.832 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:38:33.830991+0000 2024-03-29T11:38:33.832 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:33.832 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:33.832 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:33.832 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:38:33.832 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:38:33.832 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.831+0000 7f72a3323640 1 Processor -- start 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.831+0000 7f72a3323640 1 -- start start 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.831+0000 7f72a3323640 10 monclient: init 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 10 monclient(hunting): picked mon.noname-c con 0x7f729c0a9ac0 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 10 monclient(hunting): picked mon.noname-b con 0x7f729c13ee90 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 10 monclient(hunting): picked mon.noname-a con 0x7f729c0af900 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c0afd10 con 0x7f729c0af900 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c0af6f0 con 0x7f729c0a9ac0 2024-03-29T11:38:33.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c0aac80 con 0x7f729c13ee90 2024-03-29T11:38:33.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:33.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.832+0000 7f72a3323640 10 monclient(hunting): authenticate will time out at 1103.140426s 2024-03-29T11:38:33.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.835+0000 7f72a10bf640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f729c0a9ac0 0x7f729c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:49332/0 (socket says 172.21.15.145:49332) 2024-03-29T11:38:33.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.835+0000 7f72a10bf640 1 -- 172.21.15.145:0/8165230 learned_addr learned my addr 172.21.15.145:0/8165230 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.835+0000 7f7293fff640 1 -- 172.21.15.145:0/8165230 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597161651 0 0) 0x7f729c0af6f0 con 0x7f729c0a9ac0 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.835+0000 7f7293fff640 10 monclient(hunting): _negotiate 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.835+0000 7f7293fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.835+0000 7f7293fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.835+0000 7f7293fff640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.835+0000 7f7293fff640 1 -- 172.21.15.145:0/8165230 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7288003520 con 0x7f729c0a9ac0 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 1 -- 172.21.15.145:0/8165230 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 611874428 0 0) 0x7f7288003520 con 0x7f729c0a9ac0 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 1 -- 172.21.15.145:0/8165230 >> v1:172.21.15.145:6790/0 conn(0x7f729c13ee90 legacy=0x7f729c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 1 -- 172.21.15.145:0/8165230 >> v1:172.21.15.145:6789/0 conn(0x7f729c0af900 legacy=0x7f729c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: _finish_hunting 0 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 1 monclient: found mon.noname-c 2024-03-29T11:38:33.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 1 -- 172.21.15.145:0/8165230 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c0aaf60 con 0x7f729c0a9ac0 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: _finish_auth 0 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: _check_auth_tickets 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 1 -- 172.21.15.145:0/8165230 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f728c001690 con 0x7f729c0a9ac0 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: dump: 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:33.838 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 1 -- 172.21.15.145:0/8165230 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f728c001920 con 0x7f729c0a9ac0 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 1 -- 172.21.15.145:0/8165230 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f728c001cf0 con 0x7f729c0a9ac0 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f7293fff640 10 monclient: dump: 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:33.839 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f72a3323640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f72a3323640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:33.840 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.836+0000 7f72a3323640 4 set_mon_vals no callback set 2024-03-29T11:38:33.841 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.837+0000 7f72a3323640 10 monclient: shutdown 2024-03-29T11:38:33.841 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.837+0000 7f72a3323640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:33.841 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.837+0000 7f72a3323640 1 -- 172.21.15.145:0/8165230 >> v1:172.21.15.203:6789/0 conn(0x7f729c0a9ac0 legacy=0x7f729c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.841 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.837+0000 7f72a3323640 1 -- 172.21.15.145:0/8165230 shutdown_connections 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.837+0000 7f72a3323640 1 -- 172.21.15.145:0/8165230 >> 172.21.15.145:0/8165230 conn(0x7f729c0b35e0 msgr2=0x7f729c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.837+0000 7f72a3323640 1 -- 172.21.15.145:0/8165230 shutdown_connections 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.837+0000 7f72a3323640 1 -- 172.21.15.145:0/8165230 wait complete. 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.838+0000 7f72a3323640 10 monclient: build_initial_monmap 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.838+0000 7f72a3323640 10 monclient: monmap: 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:33.842 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:38:33.843 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:38:33.843 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:33.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.838+0000 7f72a3323640 1 librados: starting msgr at 2024-03-29T11:38:33.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.838+0000 7f72a3323640 1 librados: starting objecter 2024-03-29T11:38:33.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.839+0000 7f72a3323640 1 Processor -- start 2024-03-29T11:38:33.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.839+0000 7f72a3323640 1 -- start start 2024-03-29T11:38:33.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.839+0000 7f72a3323640 1 librados: setting wanted keys 2024-03-29T11:38:33.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.839+0000 7f72a3323640 1 librados: calling monclient init 2024-03-29T11:38:33.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.839+0000 7f72a3323640 10 monclient: init 2024-03-29T11:38:33.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.839+0000 7f72a3323640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:33.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.839+0000 7f72a3323640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:33.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a3323640 10 monclient(hunting): picked mon.noname-a con 0x7f729c0a9ac0 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:33.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a3323640 10 monclient(hunting): picked mon.noname-b con 0x7f729c0af900 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:33.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a3323640 10 monclient(hunting): picked mon.noname-c con 0x7f729c13ee90 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:33.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a3323640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c134e60 con 0x7f729c0a9ac0 2024-03-29T11:38:33.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a3323640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c133fc0 con 0x7f729c0af900 2024-03-29T11:38:33.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a3323640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f729c12f080 con 0x7f729c13ee90 2024-03-29T11:38:33.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a3323640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:33.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a3323640 10 monclient(hunting): authenticate will time out at 1103.151415s 2024-03-29T11:38:33.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a10bf640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f729c0a9ac0 0x7f729c134750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:46667/0 (socket says 172.21.15.145:46667) 2024-03-29T11:38:33.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.843+0000 7f72a10bf640 1 -- 172.21.15.145:0/3102037234 learned_addr learned my addr 172.21.15.145:0/3102037234 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:33.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.844+0000 7f7291ffb640 10 client.?.objecter ms_handle_connect 0x7f729c0a9ac0 2024-03-29T11:38:33.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.844+0000 7f7291ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:33.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.844+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787060387 0 0) 0x7f729c134e60 con 0x7f729c0a9ac0 2024-03-29T11:38:33.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.844+0000 7f7291ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:38:33.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.844+0000 7f7291ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:33.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.844+0000 7f7291ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:33.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.844+0000 7f7291ffb640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.844+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7280004eb0 con 0x7f729c0a9ac0 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.845+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1416492792 0 0) 0x7f7280004eb0 con 0x7f729c0a9ac0 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.845+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f729c134e60 con 0x7f729c0a9ac0 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.845+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f728c001650 con 0x7f729c0a9ac0 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.845+0000 7f7291ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.845+0000 7f7291ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.845+0000 7f7291ffb640 10 monclient(hunting): dump: 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:33.847 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:33.848 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:33.848 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:33.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.845+0000 7f7291ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:33.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.845+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 283645785 0 0) 0x7f729c134e60 con 0x7f729c0a9ac0 2024-03-29T11:38:33.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.845+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 >> v1:172.21.15.145:6790/0 conn(0x7f729c13ee90 legacy=0x7f729c1d61b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:38:33.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 >> v1:172.21.15.203:6789/0 conn(0x7f729c0af900 legacy=0x7f729c12e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:33.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f7291ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:38:33.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f7291ffb640 1 monclient: found mon.a 2024-03-29T11:38:33.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f7291ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:33.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f7291ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:33.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f729c13d540 con 0x7f729c0a9ac0 2024-03-29T11:38:33.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f7291ffb640 10 monclient: _finish_auth 0 2024-03-29T11:38:33.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f7291ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:38:33.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f7291ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f72a3323640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f72a3323640 10 monclient: _renew_subs 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f72a3323640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.846+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729c13d790 con 0x7f729c0a9ac0 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f72a3323640 10 client.4359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f72a3323640 10 monclient: _renew_subs 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f72a3323640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f729c1d6aa0 con 0x7f729c0a9ac0 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f72a3323640 1 librados: init done 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f7291ffb640 10 client.4359.objecter ms_handle_connect 0x7f729c0af900 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f7291ffb640 10 client.4359.objecter resend_mon_ops 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f728c004ca0 con 0x7f729c0a9ac0 2024-03-29T11:38:33.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f7291ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f728c005030 con 0x7f729c0a9ac0 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f7291ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f7291ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.847+0000 7f7291ffb640 10 monclient: dump: 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:33.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.849+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f728c005190 con 0x7f729c0a9ac0 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.849+0000 7f72937fe640 4 set_mon_vals no callback set 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.850+0000 7f72a08be640 10 monclient: get_auth_request con 0x7f728006dc00 auth_method 0 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.850+0000 7f7291ffb640 10 client.4359.objecter ms_handle_connect 0x7f728006dc00 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.850+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f728c084d20 con 0x7f729c0a9ac0 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.850+0000 7f7291ffb640 10 client.4359.objecter ms_dispatch 0x7f729c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.850+0000 7f7291ffb640 3 client.4359.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.850+0000 7f72a3323640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.850+0000 7f72a3323640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.850+0000 7f72a3323640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.850+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f727c005080 con 0x7f729c0a9ac0 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.851+0000 7f7291ffb640 3 client.4359.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:33.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.851+0000 7f7291ffb640 20 client.4359.objecter dump_active .. 0 homeless 2024-03-29T11:38:33.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.860+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f728c050600 con 0x7f729c0a9ac0 2024-03-29T11:38:33.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.860+0000 7f7291ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:33.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:33.860+0000 7f7291ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:34.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.180+0000 7f72a3323640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:34.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.180+0000 7f72a3323640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:34.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.180+0000 7f72a3323640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:34.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.180+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f727c005330 con 0x7f729c0a9ac0 2024-03-29T11:38:34.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.181+0000 7f7291ffb640 1 -- 172.21.15.145:0/3102037234 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f728c052530 con 0x7f729c0a9ac0 2024-03-29T11:38:34.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.181+0000 7f7291ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:34.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.181+0000 7f7291ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:34.184 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:38:34.185 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.187+0000 7f72a3323640 10 librados: watch_flush enter 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.188+0000 7f72a3323640 10 librados: watch_flush exit 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.188+0000 7f72a3323640 20 client.4359.objecter shutdown clearing up homeless session... 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.188+0000 7f72a3323640 10 client.4359.objecter successfully canceled tick 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.188+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 >> v1:172.21.15.145:6816/227372239 conn(0x7f728006dc00 legacy=0x7f72800700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.188+0000 7f72a3323640 10 monclient: shutdown 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.188+0000 7f72a3323640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.188+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 >> v1:172.21.15.145:6789/0 conn(0x7f729c0a9ac0 legacy=0x7f729c134750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.188+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 shutdown_connections 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.188+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 >> 172.21.15.145:0/3102037234 conn(0x7f729c0b35e0 msgr2=0x7f729c13ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:34.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.189+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 shutdown_connections 2024-03-29T11:38:34.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.189+0000 7f72a3323640 1 -- 172.21.15.145:0/3102037234 wait complete. 2024-03-29T11:38:34.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:34.189+0000 7f72a3323640 1 librados: shutdown 2024-03-29T11:38:34.207 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:41.013 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.156+0000 7f30248d9640 10 monclient: get_monmap_and_config 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.156+0000 7f30248d9640 10 monclient: build_initial_monmap 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.157+0000 7f30248d9640 10 monclient: monmap: 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:38:41.158275+0000 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:38:41.158275+0000 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:38:41.158 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:41.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.159+0000 7f30248d9640 1 Processor -- start 2024-03-29T11:38:41.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.159+0000 7f30248d9640 1 -- start start 2024-03-29T11:38:41.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.159+0000 7f30248d9640 10 monclient: init 2024-03-29T11:38:41.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.160+0000 7f30248d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:41.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.160+0000 7f30248d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.160+0000 7f30248d9640 10 monclient(hunting): picked mon.noname-c con 0x7f30200b4790 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f30248d9640 10 monclient(hunting): picked mon.noname-b con 0x7f30200aea90 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f30248d9640 10 monclient(hunting): picked mon.noname-a con 0x7f30200b39f0 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f30248d9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30200b0c60 con 0x7f30200b39f0 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f30248d9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30200af020 con 0x7f30200b4790 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f30248d9640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302013fbd0 con 0x7f30200aea90 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f30248d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f30248d9640 10 monclient(hunting): authenticate will time out at 1110.469090s 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f301e59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f30200b4790 0x7f30200b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:36120/0 (socket says 172.21.15.145:36120) 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f301e59c640 1 -- 172.21.15.145:0/944475041 learned_addr learned my addr 172.21.15.145:0/944475041 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f301d59a640 1 -- 172.21.15.145:0/944475041 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760871379 0 0) 0x7f30200af020 con 0x7f30200b4790 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f301d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f301d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f301d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:41.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f301d59a640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:38:41.165 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.161+0000 7f301d59a640 1 -- 172.21.15.145:0/944475041 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3008003520 con 0x7f30200b4790 2024-03-29T11:38:41.165 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.162+0000 7f301d59a640 1 -- 172.21.15.145:0/944475041 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4223538263 0 0) 0x7f3008003520 con 0x7f30200b4790 2024-03-29T11:38:41.165 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.162+0000 7f301d59a640 1 -- 172.21.15.145:0/944475041 >> v1:172.21.15.145:6790/0 conn(0x7f30200aea90 legacy=0x7f30200af540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:41.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 1 -- 172.21.15.145:0/944475041 >> v1:172.21.15.145:6789/0 conn(0x7f30200b39f0 legacy=0x7f302013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:38:41.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 1 monclient: found mon.noname-c 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 1 -- 172.21.15.145:0/944475041 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302013fd60 con 0x7f30200b4790 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 10 monclient: _finish_auth 0 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 1 -- 172.21.15.145:0/944475041 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3014001690 con 0x7f30200b4790 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f301d59a640 10 monclient: dump: 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:41.168 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f30248d9640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.164+0000 7f30248d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.165+0000 7f301d59a640 1 -- 172.21.15.145:0/944475041 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30140018f0 con 0x7f30200b4790 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.165+0000 7f301d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.165+0000 7f30248d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.165+0000 7f30248d9640 4 set_mon_vals no callback set 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.165+0000 7f301d59a640 1 -- 172.21.15.145:0/944475041 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3014001d40 con 0x7f30200b4790 2024-03-29T11:38:41.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.165+0000 7f301d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.165+0000 7f301d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.165+0000 7f301d59a640 10 monclient: dump: 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.166+0000 7f30248d9640 10 monclient: shutdown 2024-03-29T11:38:41.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.166+0000 7f30248d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:41.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.166+0000 7f30248d9640 1 -- 172.21.15.145:0/944475041 >> v1:172.21.15.203:6789/0 conn(0x7f30200b4790 legacy=0x7f30200b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:41.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.168+0000 7f30248d9640 1 -- 172.21.15.145:0/944475041 shutdown_connections 2024-03-29T11:38:41.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.168+0000 7f30248d9640 1 -- 172.21.15.145:0/944475041 >> 172.21.15.145:0/944475041 conn(0x7f3020066a00 msgr2=0x7f3020066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.169+0000 7f30248d9640 1 -- 172.21.15.145:0/944475041 shutdown_connections 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.169+0000 7f30248d9640 1 -- 172.21.15.145:0/944475041 wait complete. 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.170+0000 7f30248d9640 10 monclient: build_initial_monmap 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.170+0000 7f30248d9640 10 monclient: monmap: 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:38:41.172 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:38:41.173 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:41.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.170+0000 7f30248d9640 1 librados: starting msgr at 2024-03-29T11:38:41.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.170+0000 7f30248d9640 1 librados: starting objecter 2024-03-29T11:38:41.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.170+0000 7f30248d9640 1 Processor -- start 2024-03-29T11:38:41.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.170+0000 7f30248d9640 1 -- start start 2024-03-29T11:38:41.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.170+0000 7f30248d9640 1 librados: setting wanted keys 2024-03-29T11:38:41.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.170+0000 7f30248d9640 1 librados: calling monclient init 2024-03-29T11:38:41.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.170+0000 7f30248d9640 10 monclient: init 2024-03-29T11:38:41.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 10 monclient(hunting): picked mon.noname-b con 0x7f30200b39f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 10 monclient(hunting): picked mon.noname-a con 0x7f30200b4790 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 10 monclient(hunting): picked mon.noname-c con 0x7f3020141870 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3020131280 con 0x7f30200b4790 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3020142830 con 0x7f30200b39f0 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30201404b0 con 0x7f3020141870 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f301e59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f30200b39f0 0x7f30201e7500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:36206/0 (socket says 172.21.15.145:36206) 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f301e59c640 1 -- 172.21.15.145:0/271313700 learned_addr learned my addr 172.21.15.145:0/271313700 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:41.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.171+0000 7f30248d9640 10 monclient(hunting): authenticate will time out at 1110.479817s 2024-03-29T11:38:41.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.174+0000 7f300effd640 10 client.?.objecter ms_handle_connect 0x7f30200b39f0 2024-03-29T11:38:41.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:41.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 client.?.objecter ms_handle_connect 0x7f30200b4790 2024-03-29T11:38:41.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 client.?.objecter ms_handle_connect 0x7f3020141870 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845568726 0 0) 0x7f3020142830 con 0x7f30200b39f0 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3000004eb0 con 0x7f30200b39f0 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990427824 0 0) 0x7f3020131280 con 0x7f30200b4790 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:38:41.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3020142830 con 0x7f30200b4790 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234730826 0 0) 0x7f30201404b0 con 0x7f3020141870 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.175+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3020131280 con 0x7f3020141870 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 369359390 0 0) 0x7f3000004eb0 con 0x7f30200b39f0 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30201404b0 con 0x7f30200b39f0 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3014001650 con 0x7f30200b39f0 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): dump: 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:41.186 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4262524795 0 0) 0x7f3020142830 con 0x7f30200b4790 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3000004eb0 con 0x7f30200b4790 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1545758263 0 0) 0x7f3020131280 con 0x7f3020141870 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3020142830 con 0x7f3020141870 2024-03-29T11:38:41.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3018001a90 con 0x7f30200b4790 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): dump: 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3010001490 con 0x7f3020141870 2024-03-29T11:38:41.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient(hunting): dump: 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 991761106 0 0) 0x7f30201404b0 con 0x7f30200b39f0 2024-03-29T11:38:41.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 >> v1:172.21.15.145:6790/0 conn(0x7f3020141870 legacy=0x7f30201ef840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 >> v1:172.21.15.145:6789/0 conn(0x7f30200b4790 legacy=0x7f30201e3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 monclient: found mon.b 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302013ff70 con 0x7f30200b39f0 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient: _finish_auth 0 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f300effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f30248d9640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f30248d9640 10 monclient: _renew_subs 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f30248d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3020141250 con 0x7f30200b39f0 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f30248d9640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f30248d9640 10 monclient: _renew_subs 2024-03-29T11:38:41.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f30248d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30201e3970 con 0x7f30200b39f0 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.176+0000 7f30248d9640 1 librados: init done 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.178+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3014004a90 con 0x7f30200b39f0 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.178+0000 7f300effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.178+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3014004e20 con 0x7f30200b39f0 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.178+0000 7f300effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.179+0000 7f300effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.180+0000 7f300effd640 10 monclient: dump: 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:41.191 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.180+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f3014005040 con 0x7f30200b39f0 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.180+0000 7f301cd99640 4 set_mon_vals no callback set 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.180+0000 7f30248d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.181+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f3014084b90 con 0x7f30200b39f0 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.181+0000 7f30248d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.181+0000 7f30248d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.181+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3020067fc0 con 0x7f30200b39f0 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.181+0000 7f301dd9b640 10 monclient: get_auth_request con 0x7f300006e1b0 auth_method 0 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.181+0000 7f300effd640 10 client.4360.objecter ms_dispatch 0x7f3020000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.182+0000 7f300effd640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.182+0000 7f300effd640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.182+0000 7f300effd640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:38:41.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.182+0000 7f300effd640 10 client.4360.objecter ms_handle_connect 0x7f300006e1b0 2024-03-29T11:38:41.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.188+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3020067fc0 con 0x7f30200b39f0 2024-03-29T11:38:41.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.188+0000 7f300effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:41.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.188+0000 7f300effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:41.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.503+0000 7f30248d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:41.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.503+0000 7f30248d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:41.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.503+0000 7f30248d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:38:41.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.503+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3020067fc0 con 0x7f30200b39f0 2024-03-29T11:38:41.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.503+0000 7f300effd640 1 -- 172.21.15.145:0/271313700 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f3020067fc0 con 0x7f30200b39f0 2024-03-29T11:38:41.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.503+0000 7f300effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:41.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.503+0000 7f300effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:41.505 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:38:41.505 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.507+0000 7f30248d9640 10 librados: watch_flush enter 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.507+0000 7f30248d9640 10 librados: watch_flush exit 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.507+0000 7f30248d9640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.507+0000 7f30248d9640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.508+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 >> v1:172.21.15.145:6816/227372239 conn(0x7f300006e1b0 legacy=0x7f3000070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.508+0000 7f30248d9640 10 monclient: shutdown 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.508+0000 7f30248d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.508+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 >> v1:172.21.15.203:6789/0 conn(0x7f30200b39f0 legacy=0x7f30201e7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.508+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 shutdown_connections 2024-03-29T11:38:41.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.508+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 >> 172.21.15.145:0/271313700 conn(0x7f3020066a00 msgr2=0x7f30200b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:41.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.508+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 shutdown_connections 2024-03-29T11:38:41.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.508+0000 7f30248d9640 1 -- 172.21.15.145:0/271313700 wait complete. 2024-03-29T11:38:41.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:41.508+0000 7f30248d9640 1 librados: shutdown 2024-03-29T11:38:41.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:48.332 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:48.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.480+0000 7f76f241e640 10 monclient: get_monmap_and_config 2024-03-29T11:38:48.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.480+0000 7f76f241e640 10 monclient: build_initial_monmap 2024-03-29T11:38:48.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.480+0000 7f76f241e640 10 monclient: monmap: 2024-03-29T11:38:48.481 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:48.481 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:48.481 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:38:48.481330+0000 2024-03-29T11:38:48.481 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:38:48.481330+0000 2024-03-29T11:38:48.481 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:48.481 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:48.482 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:48.482 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:38:48.482 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:38:48.482 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:48.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.481+0000 7f76f241e640 1 Processor -- start 2024-03-29T11:38:48.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.481+0000 7f76f241e640 1 -- start start 2024-03-29T11:38:48.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.481+0000 7f76f241e640 10 monclient: init 2024-03-29T11:38:48.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:48.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:38:48.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 10 monclient(hunting): picked mon.noname-b con 0x7f76ec1457f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:48.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 10 monclient(hunting): picked mon.noname-c con 0x7f76ec13f650 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:48.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 10 monclient(hunting): picked mon.noname-a con 0x7f76ec0a9d70 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:48.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec0aaef0 con 0x7f76ec0a9d70 2024-03-29T11:38:48.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec145bd0 con 0x7f76ec13f650 2024-03-29T11:38:48.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec145eb0 con 0x7f76ec1457f0 2024-03-29T11:38:48.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:48.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.482+0000 7f76f241e640 10 monclient(hunting): authenticate will time out at 1117.790407s 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.485+0000 7f76f09bb640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f76ec0a9d70 0x7f76ec14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:43613/0 (socket says 172.21.15.145:43613) 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.485+0000 7f76f09bb640 1 -- 172.21.15.145:0/4254881525 learned_addr learned my addr 172.21.15.145:0/4254881525 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.485+0000 7f76ebfff640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f76ec1457f0 0x7f76ec13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:48545/0 (socket says 172.21.15.145:48545) 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.485+0000 7f76eb7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f76ec13f650 0x7f76ec0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:35526/0 (socket says 172.21.15.145:35526) 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858699130 0 0) 0x7f76ec0aaef0 con 0x7f76ec0a9d70 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d4003520 con 0x7f76ec0a9d70 2024-03-29T11:38:48.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292180759 0 0) 0x7f76ec145eb0 con 0x7f76ec1457f0 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ec0aaef0 con 0x7f76ec1457f0 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.486+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 597045576 0 0) 0x7f76d4003520 con 0x7f76ec0a9d70 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 >> v1:172.21.15.145:6790/0 conn(0x7f76ec1457f0 legacy=0x7f76ec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 >> v1:172.21.15.203:6789/0 conn(0x7f76ec13f650 legacy=0x7f76ec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 1 monclient: found mon.noname-a 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec146340 con 0x7f76ec0a9d70 2024-03-29T11:38:48.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f76e4001a10 con 0x7f76ec0a9d70 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: dump: 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:48.492 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76f241e640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76f241e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76e4001b70 con 0x7f76ec0a9d70 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76f241e640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76f241e640 4 set_mon_vals no callback set 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 1 -- 172.21.15.145:0/4254881525 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f76e4001f80 con 0x7f76ec0a9d70 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.487+0000 7f76eaffd640 10 monclient: dump: 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:48.493 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.488+0000 7f76f241e640 10 monclient: shutdown 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.488+0000 7f76f241e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.488+0000 7f76f241e640 1 -- 172.21.15.145:0/4254881525 >> v1:172.21.15.145:6789/0 conn(0x7f76ec0a9d70 legacy=0x7f76ec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.488+0000 7f76f241e640 1 -- 172.21.15.145:0/4254881525 shutdown_connections 2024-03-29T11:38:48.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.488+0000 7f76f241e640 1 -- 172.21.15.145:0/4254881525 >> 172.21.15.145:0/4254881525 conn(0x7f76ec066a00 msgr2=0x7f76ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.489+0000 7f76f241e640 1 -- 172.21.15.145:0/4254881525 shutdown_connections 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.489+0000 7f76f241e640 1 -- 172.21.15.145:0/4254881525 wait complete. 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.490+0000 7f76f241e640 10 monclient: build_initial_monmap 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.490+0000 7f76f241e640 10 monclient: monmap: 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:38:48.496 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.491+0000 7f76f241e640 1 librados: starting msgr at 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.491+0000 7f76f241e640 1 librados: starting objecter 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.491+0000 7f76f241e640 1 Processor -- start 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.491+0000 7f76f241e640 1 -- start start 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.491+0000 7f76f241e640 1 librados: setting wanted keys 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.491+0000 7f76f241e640 1 librados: calling monclient init 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.491+0000 7f76f241e640 10 monclient: init 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 10 monclient(hunting): picked mon.noname-b con 0x7f76ec0a9d70 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 10 monclient(hunting): picked mon.noname-a con 0x7f76ec13f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 10 monclient(hunting): picked mon.noname-c con 0x7f76ec1457f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec144bc0 con 0x7f76ec13f650 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec137f20 con 0x7f76ec0a9d70 2024-03-29T11:38:48.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ec140f40 con 0x7f76ec1457f0 2024-03-29T11:38:48.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:48.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76f241e640 10 monclient(hunting): authenticate will time out at 1117.800823s 2024-03-29T11:38:48.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76ebfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f76ec0a9d70 0x7f76ec1dee10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:35586/0 (socket says 172.21.15.145:35586) 2024-03-29T11:38:48.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.492+0000 7f76ebfff640 1 -- 172.21.15.145:0/714045534 learned_addr learned my addr 172.21.15.145:0/714045534 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:48.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 client.?.objecter ms_handle_connect 0x7f76ec0a9d70 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 client.?.objecter ms_handle_connect 0x7f76ec13f650 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793631665 0 0) 0x7f76ec137f20 con 0x7f76ec0a9d70 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76d8004eb0 con 0x7f76ec0a9d70 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024930038 0 0) 0x7f76ec144bc0 con 0x7f76ec13f650 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:48.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76ec137f20 con 0x7f76ec13f650 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3414201958 0 0) 0x7f76ec137f20 con 0x7f76ec13f650 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.496+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76ec144bc0 con 0x7f76ec13f650 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f76dc001860 con 0x7f76ec13f650 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient(hunting): dump: 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:48.502 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 121582729 0 0) 0x7f76d8004eb0 con 0x7f76ec0a9d70 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f76ec137f20 con 0x7f76ec0a9d70 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 client.?.objecter ms_handle_connect 0x7f76ec1457f0 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3288374752 0 0) 0x7f76ec144bc0 con 0x7f76ec13f650 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 >> v1:172.21.15.145:6790/0 conn(0x7f76ec1457f0 legacy=0x7f76ec137810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 >> v1:172.21.15.203:6789/0 conn(0x7f76ec0a9d70 legacy=0x7f76ec1dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 monclient: found mon.a 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:48.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ec141220 con 0x7f76ec13f650 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76f241e640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76f241e640 10 monclient: _renew_subs 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76f241e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ec13b8f0 con 0x7f76ec13f650 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76f241e640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76f241e640 10 monclient: _renew_subs 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76f241e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ec1df520 con 0x7f76ec13f650 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76f241e640 1 librados: init done 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f76dc0047e0 con 0x7f76ec13f650 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f76dc004b30 con 0x7f76ec13f650 2024-03-29T11:38:48.504 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76e8ff9640 10 monclient: dump: 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.497+0000 7f76ea7fc640 4 set_mon_vals no callback set 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.498+0000 7f76f241e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.498+0000 7f76f241e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.498+0000 7f76f241e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:48.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.498+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76cc005080 con 0x7f76ec13f650 2024-03-29T11:38:48.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.500+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f76dc01ab00 con 0x7f76ec13f650 2024-03-29T11:38:48.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.501+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f76dc084920 con 0x7f76ec13f650 2024-03-29T11:38:48.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.501+0000 7f76e8ff9640 10 client.4368.objecter ms_dispatch 0x7f76ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:48.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.501+0000 7f76e8ff9640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:48.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.501+0000 7f76e8ff9640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:48.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.501+0000 7f76e8ff9640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:38:48.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.502+0000 7f76ebfff640 10 monclient: get_auth_request con 0x7f76d806de90 auth_method 0 2024-03-29T11:38:48.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.502+0000 7f76e8ff9640 10 client.4368.objecter ms_handle_connect 0x7f76d806de90 2024-03-29T11:38:48.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.507+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f76dc050200 con 0x7f76ec13f650 2024-03-29T11:38:48.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.507+0000 7f76e8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:48.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.507+0000 7f76e8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:48.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.829+0000 7f76f241e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:48.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.830+0000 7f76f241e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:48.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.830+0000 7f76f241e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:38:48.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.830+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76cc005330 con 0x7f76ec13f650 2024-03-29T11:38:48.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.831+0000 7f76e8ff9640 1 -- 172.21.15.145:0/714045534 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f76dc052130 con 0x7f76ec13f650 2024-03-29T11:38:48.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.832+0000 7f76e8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:48.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.832+0000 7f76e8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:48.833 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:38:48.833 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:48.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.835+0000 7f76f241e640 10 librados: watch_flush enter 2024-03-29T11:38:48.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.836+0000 7f76f241e640 10 librados: watch_flush exit 2024-03-29T11:38:48.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.836+0000 7f76f241e640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:38:48.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.836+0000 7f76f241e640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:38:48.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.836+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 >> v1:172.21.15.145:6816/227372239 conn(0x7f76d806de90 legacy=0x7f76d8070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.836+0000 7f76f241e640 10 monclient: shutdown 2024-03-29T11:38:48.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.836+0000 7f76f241e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:48.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.836+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 >> v1:172.21.15.145:6789/0 conn(0x7f76ec13f650 legacy=0x7f76ec137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:48.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.837+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 shutdown_connections 2024-03-29T11:38:48.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.837+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 >> 172.21.15.145:0/714045534 conn(0x7f76ec066a00 msgr2=0x7f76ec066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:48.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.837+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 shutdown_connections 2024-03-29T11:38:48.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.837+0000 7f76f241e640 1 -- 172.21.15.145:0/714045534 wait complete. 2024-03-29T11:38:48.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:48.837+0000 7f76f241e640 1 librados: shutdown 2024-03-29T11:38:48.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:38:55.661 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:38:55.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.806+0000 7efe59f53640 10 monclient: get_monmap_and_config 2024-03-29T11:38:55.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.806+0000 7efe59f53640 10 monclient: build_initial_monmap 2024-03-29T11:38:55.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.806+0000 7efe59f53640 10 monclient: monmap: 2024-03-29T11:38:55.807 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:55.807 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:55.807 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:38:55.807477+0000 2024-03-29T11:38:55.808 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:38:55.807477+0000 2024-03-29T11:38:55.808 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:55.808 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:55.808 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:55.808 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:38:55.808 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:38:55.808 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:55.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.807+0000 7efe59f53640 1 Processor -- start 2024-03-29T11:38:55.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.807+0000 7efe59f53640 1 -- start start 2024-03-29T11:38:55.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.807+0000 7efe59f53640 10 monclient: init 2024-03-29T11:38:55.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:55.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:38:55.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 10 monclient(hunting): picked mon.noname-a con 0x7efe4c006940 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:55.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 10 monclient(hunting): picked mon.noname-b con 0x7efe4c035480 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:55.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 10 monclient(hunting): picked mon.noname-c con 0x7efe4c0c4ec0 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:55.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c036f30 con 0x7efe4c006940 2024-03-29T11:38:55.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c037210 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c034f60 con 0x7efe4c035480 2024-03-29T11:38:55.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:55.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.808+0000 7efe59f53640 10 monclient(hunting): authenticate will time out at 1125.116985s 2024-03-29T11:38:55.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.810+0000 7efe59752640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7efe4c0c4ec0 0x7efe4c0d30a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:45410/0 (socket says 172.21.15.145:45410) 2024-03-29T11:38:55.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.810+0000 7efe59752640 1 -- 172.21.15.145:0/3732384335 learned_addr learned my addr 172.21.15.145:0/3732384335 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:55.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.813+0000 7efe537fe640 1 -- 172.21.15.145:0/3732384335 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474903070 0 0) 0x7efe4c036f30 con 0x7efe4c006940 2024-03-29T11:38:55.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.813+0000 7efe537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:38:55.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.813+0000 7efe537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:55.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.813+0000 7efe537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:55.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.813+0000 7efe537fe640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:38:55.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.813+0000 7efe537fe640 1 -- 172.21.15.145:0/3732384335 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe44003520 con 0x7efe4c006940 2024-03-29T11:38:55.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.813+0000 7efe537fe640 1 -- 172.21.15.145:0/3732384335 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1593679150 0 0) 0x7efe44003520 con 0x7efe4c006940 2024-03-29T11:38:55.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.813+0000 7efe537fe640 1 -- 172.21.15.145:0/3732384335 >> v1:172.21.15.145:6790/0 conn(0x7efe4c035480 legacy=0x7efe4c0c47b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:38:55.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 1 -- 172.21.15.145:0/3732384335 >> v1:172.21.15.203:6789/0 conn(0x7efe4c0c4ec0 legacy=0x7efe4c0d30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:38:55.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 1 monclient: found mon.noname-a 2024-03-29T11:38:55.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:55.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 1 -- 172.21.15.145:0/3732384335 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c0cb410 con 0x7efe4c006940 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 10 monclient: _finish_auth 0 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 1 -- 172.21.15.145:0/3732384335 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe5404f730 con 0x7efe4c006940 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.815+0000 7efe537fe640 10 monclient: dump: 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:55.818 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.816+0000 7efe537fe640 1 -- 172.21.15.145:0/3732384335 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe540008d0 con 0x7efe4c006940 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.816+0000 7efe537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.816+0000 7efe537fe640 1 -- 172.21.15.145:0/3732384335 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe540758b0 con 0x7efe4c006940 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.816+0000 7efe537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.816+0000 7efe537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.816+0000 7efe537fe640 10 monclient: dump: 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:55.819 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:55.820 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:55.820 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:55.820 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:55.820 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:55.820 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:55.820 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.817+0000 7efe59f53640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.817+0000 7efe59f53640 10 monclient: get_monmap_and_config success 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.817+0000 7efe59f53640 4 set_mon_vals no callback set 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.817+0000 7efe59f53640 10 monclient: shutdown 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.817+0000 7efe59f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.818+0000 7efe59f53640 1 -- 172.21.15.145:0/3732384335 >> v1:172.21.15.145:6789/0 conn(0x7efe4c006940 legacy=0x7efe4c0cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.819+0000 7efe59f53640 1 -- 172.21.15.145:0/3732384335 shutdown_connections 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.819+0000 7efe59f53640 1 -- 172.21.15.145:0/3732384335 >> 172.21.15.145:0/3732384335 conn(0x7efe4c0058f0 msgr2=0x7efe4c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.819+0000 7efe59f53640 1 -- 172.21.15.145:0/3732384335 shutdown_connections 2024-03-29T11:38:55.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.819+0000 7efe59f53640 1 -- 172.21.15.145:0/3732384335 wait complete. 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.820+0000 7efe59f53640 10 monclient: build_initial_monmap 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.820+0000 7efe59f53640 10 monclient: monmap: 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.820+0000 7efe59f53640 1 librados: starting msgr at 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.820+0000 7efe59f53640 1 librados: starting objecter 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.820+0000 7efe59f53640 1 Processor -- start 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.820+0000 7efe59f53640 1 -- start start 2024-03-29T11:38:55.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.821+0000 7efe59f53640 1 librados: setting wanted keys 2024-03-29T11:38:55.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.821+0000 7efe59f53640 1 librados: calling monclient init 2024-03-29T11:38:55.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.821+0000 7efe59f53640 10 monclient: init 2024-03-29T11:38:55.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.822+0000 7efe59f53640 10 monclient: _reopen_session rank -1 2024-03-29T11:38:55.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.822+0000 7efe59f53640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:38:55.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.822+0000 7efe59f53640 10 monclient(hunting): picked mon.noname-a con 0x7efe4c006940 addr v1:172.21.15.145:6789/0 2024-03-29T11:38:55.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.822+0000 7efe59f53640 10 monclient(hunting): picked mon.noname-b con 0x7efe4c035480 addr v1:172.21.15.203:6789/0 2024-03-29T11:38:55.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.822+0000 7efe59f53640 10 monclient(hunting): picked mon.noname-c con 0x7efe4c0c4ec0 addr v1:172.21.15.145:6790/0 2024-03-29T11:38:55.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.822+0000 7efe59f53640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c0310f0 con 0x7efe4c006940 2024-03-29T11:38:55.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.823+0000 7efe59f53640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c0c0c80 con 0x7efe4c035480 2024-03-29T11:38:55.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.823+0000 7efe59f53640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c031360 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.823+0000 7efe59f53640 10 monclient(hunting): _renew_subs 2024-03-29T11:38:55.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.823+0000 7efe59f53640 10 monclient(hunting): authenticate will time out at 1125.131048s 2024-03-29T11:38:55.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe53fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7efe4c035480 0x7efe4c0bfe60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:45564/0 (socket says 172.21.15.145:45564) 2024-03-29T11:38:55.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe53fff640 1 -- 172.21.15.145:0/3108088768 learned_addr learned my addr 172.21.15.145:0/3108088768 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:38:55.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe517fa640 10 client.?.objecter ms_handle_connect 0x7efe4c035480 2024-03-29T11:38:55.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3797191503 0 0) 0x7efe4c0c0c80 con 0x7efe4c035480 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe517fa640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.826+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe38004eb0 con 0x7efe4c035480 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 client.?.objecter ms_handle_connect 0x7efe4c0c4ec0 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337759655 0 0) 0x7efe4c031360 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe38006230 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3294766781 0 0) 0x7efe38006230 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe4c031360 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe48001810 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): dump: 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:55.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 client.?.objecter ms_handle_connect 0x7efe4c006940 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420461337 0 0) 0x7efe4c0310f0 con 0x7efe4c006940 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): _negotiate 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe38006230 con 0x7efe4c006940 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2844313917 0 0) 0x7efe4c031360 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.827+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 >> v1:172.21.15.203:6789/0 conn(0x7efe4c035480 legacy=0x7efe4c0bfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 >> v1:172.21.15.145:6789/0 conn(0x7efe4c006940 legacy=0x7efe4c02fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 10 monclient: _finish_hunting 0 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 1 monclient: found mon.c 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:38:55.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c0302b0 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 10 monclient: _finish_auth 0 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 10 monclient: _check_auth_tickets 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe48001c60 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efe48004980 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe517fa640 10 monclient: dump: 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:38:55.836 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.828+0000 7efe52ffd640 4 set_mon_vals no callback set 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.829+0000 7efe59f53640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.829+0000 7efe59f53640 10 monclient: _renew_subs 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.829+0000 7efe59f53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.830+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4c02f1e0 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.830+0000 7efe59f53640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.830+0000 7efe59f53640 10 monclient: _renew_subs 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.830+0000 7efe59f53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.830+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4c032db0 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.830+0000 7efe59f53640 1 librados: init done 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.832+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7efe4801a950 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.832+0000 7efe59f53640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.832+0000 7efe59f53640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.832+0000 7efe59f53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.832+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c004cd0 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.833+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7efe48084930 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.833+0000 7efe517fa640 10 client.4373.objecter ms_dispatch 0x7efe54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.833+0000 7efe517fa640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.833+0000 7efe517fa640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.833+0000 7efe517fa640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.834+0000 7efe58f51640 10 monclient: get_auth_request con 0x7efe3806dd30 auth_method 0 2024-03-29T11:38:55.839 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.834+0000 7efe517fa640 10 client.4373.objecter ms_handle_connect 0x7efe3806dd30 2024-03-29T11:38:55.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.842+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe48050040 con 0x7efe4c0c4ec0 2024-03-29T11:38:55.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.842+0000 7efe517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:38:55.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:55.842+0000 7efe517fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:38:56.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.166+0000 7efe59f53640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:56.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.166+0000 7efe59f53640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:56.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.166+0000 7efe59f53640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:38:56.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.166+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe4c0086f0 con 0x7efe4c0c4ec0 2024-03-29T11:38:56.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.170+0000 7efe517fa640 1 -- 172.21.15.145:0/3108088768 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7efe48052140 con 0x7efe4c0c4ec0 2024-03-29T11:38:56.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.170+0000 7efe517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:38:56.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.170+0000 7efe517fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:38:56.171 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:38:56.171 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:38:56.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.173+0000 7efe59f53640 10 librados: watch_flush enter 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.173+0000 7efe59f53640 10 librados: watch_flush exit 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.173+0000 7efe59f53640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.173+0000 7efe59f53640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.174+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 >> v1:172.21.15.145:6816/227372239 conn(0x7efe3806dd30 legacy=0x7efe380701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.174+0000 7efe59f53640 10 monclient: shutdown 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.174+0000 7efe59f53640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.174+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 >> v1:172.21.15.145:6790/0 conn(0x7efe4c0c4ec0 legacy=0x7efe4c0c0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.174+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 shutdown_connections 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.174+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 >> 172.21.15.145:0/3108088768 conn(0x7efe4c0058f0 msgr2=0x7efe4c005cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.174+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 shutdown_connections 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.174+0000 7efe59f53640 1 -- 172.21.15.145:0/3108088768 wait complete. 2024-03-29T11:38:56.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:38:56.174+0000 7efe59f53640 1 librados: shutdown 2024-03-29T11:38:56.192 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:02.999 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:03.148 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.146+0000 7f363879d640 10 monclient: get_monmap_and_config 2024-03-29T11:39:03.148 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.146+0000 7f363879d640 10 monclient: build_initial_monmap 2024-03-29T11:39:03.148 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.146+0000 7f363879d640 10 monclient: monmap: 2024-03-29T11:39:03.148 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:03.148 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:03.148 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:39:03.147765+0000 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:39:03.147765+0000 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.147+0000 7f363879d640 1 Processor -- start 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.147+0000 7f363879d640 1 -- start start 2024-03-29T11:39:03.149 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.147+0000 7f363879d640 10 monclient: init 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 10 monclient(hunting): picked mon.noname-c con 0x7f36301457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 10 monclient(hunting): picked mon.noname-a con 0x7f363013f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 10 monclient(hunting): picked mon.noname-b con 0x7f36300a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36300aaef0 con 0x7f363013f650 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3630145bd0 con 0x7f36301457f0 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3630145eb0 con 0x7f36300a9d70 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:03.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.148+0000 7f363879d640 10 monclient(hunting): authenticate will time out at 1132.456547s 2024-03-29T11:39:03.151 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.150+0000 7f3636539640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f36301457f0 0x7f363013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:42674/0 (socket says 172.21.15.145:42674) 2024-03-29T11:39:03.151 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.150+0000 7f3636539640 1 -- 172.21.15.145:0/2320904450 learned_addr learned my addr 172.21.15.145:0/2320904450 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:03.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.152+0000 7f3635537640 1 -- 172.21.15.145:0/2320904450 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879422299 0 0) 0x7f36300aaef0 con 0x7f363013f650 2024-03-29T11:39:03.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.152+0000 7f3635537640 10 monclient(hunting): _negotiate 2024-03-29T11:39:03.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.152+0000 7f3635537640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:03.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.152+0000 7f3635537640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:03.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.152+0000 7f3635537640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:39:03.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.152+0000 7f3635537640 1 -- 172.21.15.145:0/2320904450 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3620003520 con 0x7f363013f650 2024-03-29T11:39:03.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.152+0000 7f3635537640 1 -- 172.21.15.145:0/2320904450 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4063708453 0 0) 0x7f3620003520 con 0x7f363013f650 2024-03-29T11:39:03.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.152+0000 7f3635537640 1 -- 172.21.15.145:0/2320904450 >> v1:172.21.15.145:6790/0 conn(0x7f36300a9d70 legacy=0x7f363014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:03.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.153+0000 7f3635537640 1 -- 172.21.15.145:0/2320904450 >> v1:172.21.15.203:6789/0 conn(0x7f36301457f0 legacy=0x7f363013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:03.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.153+0000 7f3635537640 10 monclient: _finish_hunting 0 2024-03-29T11:39:03.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.153+0000 7f3635537640 1 monclient: found mon.noname-a 2024-03-29T11:39:03.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.153+0000 7f3635537640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:03.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.153+0000 7f3635537640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:39:03.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.153+0000 7f3635537640 1 -- 172.21.15.145:0/2320904450 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3630146340 con 0x7f363013f650 2024-03-29T11:39:03.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.153+0000 7f3635537640 10 monclient: _finish_auth 0 2024-03-29T11:39:03.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.153+0000 7f3635537640 10 monclient: _check_auth_tickets 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.153+0000 7f3635537640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.154+0000 7f3635537640 1 -- 172.21.15.145:0/2320904450 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f361c001a10 con 0x7f363013f650 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.154+0000 7f3635537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.154+0000 7f3635537640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.154+0000 7f3635537640 10 monclient: dump: 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:03.157 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:03.158 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:03.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.154+0000 7f363879d640 5 monclient: authenticate success, global_id 4377 2024-03-29T11:39:03.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.154+0000 7f363879d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:03.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.157+0000 7f3635537640 1 -- 172.21.15.145:0/2320904450 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f361c001be0 con 0x7f363013f650 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.157+0000 7f3635537640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.157+0000 7f3635537640 1 -- 172.21.15.145:0/2320904450 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f361c002030 con 0x7f363013f650 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.157+0000 7f3635537640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.157+0000 7f3635537640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.157+0000 7f3635537640 10 monclient: dump: 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:03.159 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:03.160 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:03.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.157+0000 7f363879d640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:03.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.157+0000 7f363879d640 4 set_mon_vals no callback set 2024-03-29T11:39:03.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.158+0000 7f363879d640 10 monclient: shutdown 2024-03-29T11:39:03.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.158+0000 7f363879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:03.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.158+0000 7f363879d640 1 -- 172.21.15.145:0/2320904450 >> v1:172.21.15.145:6789/0 conn(0x7f363013f650 legacy=0x7f36300b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:03.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.159+0000 7f363879d640 1 -- 172.21.15.145:0/2320904450 shutdown_connections 2024-03-29T11:39:03.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.159+0000 7f363879d640 1 -- 172.21.15.145:0/2320904450 >> 172.21.15.145:0/2320904450 conn(0x7f3630066a00 msgr2=0x7f3630066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:03.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.159+0000 7f363879d640 1 -- 172.21.15.145:0/2320904450 shutdown_connections 2024-03-29T11:39:03.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.159+0000 7f363879d640 1 -- 172.21.15.145:0/2320904450 wait complete. 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.160+0000 7f363879d640 10 monclient: build_initial_monmap 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.160+0000 7f363879d640 10 monclient: monmap: 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:39:03.162 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:39:03.163 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:03.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.160+0000 7f363879d640 1 librados: starting msgr at 2024-03-29T11:39:03.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.160+0000 7f363879d640 1 librados: starting objecter 2024-03-29T11:39:03.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.161+0000 7f363879d640 1 Processor -- start 2024-03-29T11:39:03.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.161+0000 7f363879d640 1 -- start start 2024-03-29T11:39:03.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.161+0000 7f363879d640 1 librados: setting wanted keys 2024-03-29T11:39:03.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.161+0000 7f363879d640 1 librados: calling monclient init 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.161+0000 7f363879d640 10 monclient: init 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.161+0000 7f363879d640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f363879d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f363879d640 10 monclient(hunting): picked mon.noname-c con 0x7f36300a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f363879d640 10 monclient(hunting): picked mon.noname-b con 0x7f363013f650 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f363879d640 10 monclient(hunting): picked mon.noname-a con 0x7f36301457f0 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f363879d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3630143f20 con 0x7f36301457f0 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f363879d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36301391f0 con 0x7f363013f650 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f363879d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36301394d0 con 0x7f36300a9d70 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f363879d640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f363879d640 10 monclient(hunting): authenticate will time out at 1132.470127s 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f3635d38640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f363013f650 0x7f363013cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:43416/0 (socket says 172.21.15.145:43416) 2024-03-29T11:39:03.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.162+0000 7f3635d38640 1 -- 172.21.15.145:0/461549523 learned_addr learned my addr 172.21.15.145:0/461549523 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.163+0000 7f362effd640 10 client.?.objecter ms_handle_connect 0x7f363013f650 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.163+0000 7f362effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.163+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394895635 0 0) 0x7f36301391f0 con 0x7f363013f650 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.163+0000 7f362effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.163+0000 7f362effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.163+0000 7f362effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.163+0000 7f362effd640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.163+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3614004eb0 con 0x7f363013f650 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.164+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888048448 0 0) 0x7f3614004eb0 con 0x7f363013f650 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.164+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3614005300 con 0x7f363013f650 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.164+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f361c001fb0 con 0x7f363013f650 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.164+0000 7f362effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.164+0000 7f362effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:03.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.164+0000 7f362effd640 10 monclient(hunting): dump: 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.164+0000 7f362effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.165+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4290556249 0 0) 0x7f3614005300 con 0x7f363013f650 2024-03-29T11:39:03.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.165+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 >> v1:172.21.15.145:6790/0 conn(0x7f36300a9d70 legacy=0x7f3630144920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:03.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 >> v1:172.21.15.145:6789/0 conn(0x7f36301457f0 legacy=0x7f36301ee8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 1 monclient: found mon.b 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363013abe0 con 0x7f363013f650 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 monclient: _finish_auth 0 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 client.?.objecter ms_handle_connect 0x7f36301457f0 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f363879d640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f363879d640 10 monclient: _renew_subs 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f363879d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363013ae90 con 0x7f363013f650 2024-03-29T11:39:03.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f363879d640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f363879d640 10 monclient: _renew_subs 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f363879d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f363013d5f0 con 0x7f363013f650 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f363879d640 1 librados: init done 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f361c003cc0 con 0x7f363013f650 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f361c004110 con 0x7f363013f650 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f362effd640 10 monclient: dump: 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:03.170 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:03.171 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:03.171 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:03.171 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:03.171 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:03.171 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:03.171 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:03.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.166+0000 7f3634d36640 4 set_mon_vals no callback set 2024-03-29T11:39:03.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.169+0000 7f363879d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:03.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.169+0000 7f363879d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:03.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.169+0000 7f363879d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:03.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.169+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3618005080 con 0x7f363013f650 2024-03-29T11:39:03.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.169+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f361c004270 con 0x7f363013f650 2024-03-29T11:39:03.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.170+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f361c084f40 con 0x7f363013f650 2024-03-29T11:39:03.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.170+0000 7f362effd640 10 client.4375.objecter ms_dispatch 0x7f3630000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:03.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.170+0000 7f362effd640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:03.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.170+0000 7f362effd640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:03.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.170+0000 7f362effd640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T11:39:03.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.170+0000 7f3636539640 10 monclient: get_auth_request con 0x7f361406ddb0 auth_method 0 2024-03-29T11:39:03.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.170+0000 7f362effd640 10 client.4375.objecter ms_handle_connect 0x7f361406ddb0 2024-03-29T11:39:03.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.178+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f361c050820 con 0x7f363013f650 2024-03-29T11:39:03.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.178+0000 7f362effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:03.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.178+0000 7f362effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:03.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.507+0000 7f363879d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:03.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.507+0000 7f363879d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:03.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.507+0000 7f363879d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:03.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.507+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3618005330 con 0x7f363013f650 2024-03-29T11:39:03.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.508+0000 7f362effd640 1 -- 172.21.15.145:0/461549523 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f361c052750 con 0x7f363013f650 2024-03-29T11:39:03.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.508+0000 7f362effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:03.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.508+0000 7f362effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:03.510 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:39:03.510 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:03.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.512+0000 7f363879d640 10 librados: watch_flush enter 2024-03-29T11:39:03.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.512+0000 7f363879d640 10 librados: watch_flush exit 2024-03-29T11:39:03.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.512+0000 7f363879d640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T11:39:03.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.512+0000 7f363879d640 10 client.4375.objecter successfully canceled tick 2024-03-29T11:39:03.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.512+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 >> v1:172.21.15.145:6816/227372239 conn(0x7f361406ddb0 legacy=0x7f3614070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:03.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.512+0000 7f363879d640 10 monclient: shutdown 2024-03-29T11:39:03.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.512+0000 7f363879d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:03.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.512+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 >> v1:172.21.15.203:6789/0 conn(0x7f363013f650 legacy=0x7f363013cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:03.515 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.514+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 shutdown_connections 2024-03-29T11:39:03.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.514+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 >> 172.21.15.145:0/461549523 conn(0x7f3630066a00 msgr2=0x7f36300b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:03.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.514+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 shutdown_connections 2024-03-29T11:39:03.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.514+0000 7f363879d640 1 -- 172.21.15.145:0/461549523 wait complete. 2024-03-29T11:39:03.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:03.514+0000 7f363879d640 1 librados: shutdown 2024-03-29T11:39:03.533 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:10.341 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:10.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.497+0000 7f2874cff640 10 monclient: get_monmap_and_config 2024-03-29T11:39:10.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.497+0000 7f2874cff640 10 monclient: build_initial_monmap 2024-03-29T11:39:10.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.497+0000 7f2874cff640 10 monclient: monmap: 2024-03-29T11:39:10.497 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:10.497 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:10.498 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:39:10.497569+0000 2024-03-29T11:39:10.498 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:39:10.497569+0000 2024-03-29T11:39:10.498 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:10.498 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:10.498 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:10.498 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:39:10.498 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:39:10.498 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:10.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.498+0000 7f2874cff640 1 Processor -- start 2024-03-29T11:39:10.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.498+0000 7f2874cff640 1 -- start start 2024-03-29T11:39:10.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.498+0000 7f2874cff640 10 monclient: init 2024-03-29T11:39:10.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:10.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:10.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 10 monclient(hunting): picked mon.noname-c con 0x7f28700b1f60 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:10.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 10 monclient(hunting): picked mon.noname-b con 0x7f2870071eb0 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:10.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 10 monclient(hunting): picked mon.noname-a con 0x7f2870143d10 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:10.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287006ef80 con 0x7f2870143d10 2024-03-29T11:39:10.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287006f260 con 0x7f28700b1f60 2024-03-29T11:39:10.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28700b40c0 con 0x7f2870071eb0 2024-03-29T11:39:10.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:10.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.499+0000 7f2874cff640 10 monclient(hunting): authenticate will time out at 1139.806571s 2024-03-29T11:39:10.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.503+0000 7f286ed9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f2870143d10 0x7f2870146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:54421/0 (socket says 172.21.15.145:54421) 2024-03-29T11:39:10.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.503+0000 7f286ed9d640 1 -- 172.21.15.145:0/2442811304 learned_addr learned my addr 172.21.15.145:0/2442811304 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:10.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409736338 0 0) 0x7f28700b40c0 con 0x7f2870071eb0 2024-03-29T11:39:10.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2858003520 con 0x7f2870071eb0 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792941309 0 0) 0x7f287006ef80 con 0x7f2870143d10 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28700b40c0 con 0x7f2870143d10 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730866147 0 0) 0x7f287006f260 con 0x7f28700b1f60 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.506+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f287006ef80 con 0x7f28700b1f60 2024-03-29T11:39:10.508 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 72136000 0 0) 0x7f2858003520 con 0x7f2870071eb0 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 >> v1:172.21.15.203:6789/0 conn(0x7f28700b1f60 legacy=0x7f2870143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 >> v1:172.21.15.145:6789/0 conn(0x7f2870143d10 legacy=0x7f2870146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 1 monclient: found mon.noname-b 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2870155d70 con 0x7f2870071eb0 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: _finish_auth 0 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2868001a10 con 0x7f2870071eb0 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: dump: 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:10.509 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2868001b70 con 0x7f2870071eb0 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 1 -- 172.21.15.145:0/2442811304 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2868001f80 con 0x7f2870071eb0 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f286d59a640 10 monclient: dump: 2024-03-29T11:39:10.510 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:10.511 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:10.511 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:10.511 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:10.511 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:10.511 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:10.511 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:10.511 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:10.511 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:10.511 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f2874cff640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f2874cff640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.507+0000 7f2874cff640 4 set_mon_vals no callback set 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.508+0000 7f2874cff640 10 monclient: shutdown 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.508+0000 7f2874cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.508+0000 7f2874cff640 1 -- 172.21.15.145:0/2442811304 >> v1:172.21.15.145:6790/0 conn(0x7f2870071eb0 legacy=0x7f2870146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.508+0000 7f2874cff640 1 -- 172.21.15.145:0/2442811304 shutdown_connections 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.508+0000 7f2874cff640 1 -- 172.21.15.145:0/2442811304 >> 172.21.15.145:0/2442811304 conn(0x7f2870066a00 msgr2=0x7f2870066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.509+0000 7f2874cff640 1 -- 172.21.15.145:0/2442811304 shutdown_connections 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.509+0000 7f2874cff640 1 -- 172.21.15.145:0/2442811304 wait complete. 2024-03-29T11:39:10.513 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.510+0000 7f2874cff640 10 monclient: build_initial_monmap 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.510+0000 7f2874cff640 10 monclient: monmap: 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.510+0000 7f2874cff640 1 librados: starting msgr at 2024-03-29T11:39:10.514 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.510+0000 7f2874cff640 1 librados: starting objecter 2024-03-29T11:39:10.515 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.510+0000 7f2874cff640 1 Processor -- start 2024-03-29T11:39:10.515 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.511+0000 7f2874cff640 1 -- start start 2024-03-29T11:39:10.515 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.511+0000 7f2874cff640 1 librados: setting wanted keys 2024-03-29T11:39:10.515 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.511+0000 7f2874cff640 1 librados: calling monclient init 2024-03-29T11:39:10.515 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.511+0000 7f2874cff640 10 monclient: init 2024-03-29T11:39:10.515 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:10.515 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 10 monclient(hunting): picked mon.noname-b con 0x7f28700b1f60 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 10 monclient(hunting): picked mon.noname-c con 0x7f2870143d10 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 10 monclient(hunting): picked mon.noname-a con 0x7f287013f390 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2870147a30 con 0x7f287013f390 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287014ad00 con 0x7f28700b1f60 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f287014afe0 con 0x7f2870143d10 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.512+0000 7f2874cff640 10 monclient(hunting): authenticate will time out at 1139.819562s 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.515+0000 7f286dd9b640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f2870143d10 0x7f28701e2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:33355/0 (socket says 172.21.15.145:33355) 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.515+0000 7f286dd9b640 1 -- 172.21.15.145:0/4049647799 learned_addr learned my addr 172.21.15.145:0/4049647799 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.515+0000 7f285effd640 10 client.?.objecter ms_handle_connect 0x7f2870143d10 2024-03-29T11:39:10.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.515+0000 7f285effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:10.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.517+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898072049 0 0) 0x7f287014afe0 con 0x7f2870143d10 2024-03-29T11:39:10.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.518+0000 7f285effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.518+0000 7f285effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.518+0000 7f285effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.518+0000 7f285effd640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:39:10.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.518+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2850004eb0 con 0x7f2870143d10 2024-03-29T11:39:10.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.518+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2403140932 0 0) 0x7f2850004eb0 con 0x7f2870143d10 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.518+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f287014afe0 con 0x7f2870143d10 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.518+0000 7f285effd640 10 client.?.objecter ms_handle_connect 0x7f28700b1f60 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.518+0000 7f285effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2868001a10 con 0x7f2870143d10 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 10 monclient(hunting): dump: 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:10.522 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527255102 0 0) 0x7f287014ad00 con 0x7f28700b1f60 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2850004eb0 con 0x7f28700b1f60 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3165927962 0 0) 0x7f287014afe0 con 0x7f2870143d10 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 >> v1:172.21.15.203:6789/0 conn(0x7f28700b1f60 legacy=0x7f2870147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 >> v1:172.21.15.145:6789/0 conn(0x7f287013f390 legacy=0x7f28701e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.519+0000 7f285effd640 1 monclient: found mon.c 2024-03-29T11:39:10.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f285effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f285effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287014b2c0 con 0x7f2870143d10 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f285effd640 10 monclient: _finish_auth 0 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f285effd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f285effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f2874cff640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f2874cff640 10 monclient: _renew_subs 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f2874cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287013f940 con 0x7f2870143d10 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f2874cff640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f2874cff640 10 monclient: _renew_subs 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f2874cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2870154c40 con 0x7f2870143d10 2024-03-29T11:39:10.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.520+0000 7f2874cff640 1 librados: init done 2024-03-29T11:39:10.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.522+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2868003970 con 0x7f2870143d10 2024-03-29T11:39:10.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.522+0000 7f285effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.522+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2868004020 con 0x7f2870143d10 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.522+0000 7f285effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.522+0000 7f285effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.522+0000 7f285effd640 10 monclient: dump: 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.522+0000 7f286cd99640 4 set_mon_vals no callback set 2024-03-29T11:39:10.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.523+0000 7f2874cff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.524+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f2868004180 con 0x7f2870143d10 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.524+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f2868085050 con 0x7f2870143d10 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.525+0000 7f285effd640 10 client.4379.objecter ms_dispatch 0x7f2870000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.525+0000 7f285effd640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.525+0000 7f285effd640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.525+0000 7f285effd640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.525+0000 7f2874cff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.525+0000 7f2874cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.525+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2854005080 con 0x7f2870143d10 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.525+0000 7f286e59c640 10 monclient: get_auth_request con 0x7f285006de40 auth_method 0 2024-03-29T11:39:10.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.526+0000 7f285effd640 10 client.4379.objecter ms_handle_connect 0x7f285006de40 2024-03-29T11:39:10.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.534+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2868050930 con 0x7f2870143d10 2024-03-29T11:39:10.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.534+0000 7f285effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:10.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.534+0000 7f285effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:10.858 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.857+0000 7f2874cff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:10.858 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.857+0000 7f2874cff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:10.858 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.857+0000 7f2874cff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:10.858 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.857+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2854005330 con 0x7f2870143d10 2024-03-29T11:39:10.858 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.858+0000 7f285effd640 1 -- 172.21.15.145:0/4049647799 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f2868052860 con 0x7f2870143d10 2024-03-29T11:39:10.858 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.858+0000 7f285effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:10.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.858+0000 7f285effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:10.860 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:39:10.860 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:10.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.862+0000 7f2874cff640 10 librados: watch_flush enter 2024-03-29T11:39:10.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.862+0000 7f2874cff640 10 librados: watch_flush exit 2024-03-29T11:39:10.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.862+0000 7f2874cff640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-29T11:39:10.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.862+0000 7f2874cff640 10 client.4379.objecter successfully canceled tick 2024-03-29T11:39:10.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.862+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 >> v1:172.21.15.145:6816/227372239 conn(0x7f285006de40 legacy=0x7f2850070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.862+0000 7f2874cff640 10 monclient: shutdown 2024-03-29T11:39:10.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.862+0000 7f2874cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:10.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.862+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 >> v1:172.21.15.145:6790/0 conn(0x7f2870143d10 legacy=0x7f28701e2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:10.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.863+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 shutdown_connections 2024-03-29T11:39:10.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.863+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 >> 172.21.15.145:0/4049647799 conn(0x7f2870066a00 msgr2=0x7f2870142860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:10.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.863+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 shutdown_connections 2024-03-29T11:39:10.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.863+0000 7f2874cff640 1 -- 172.21.15.145:0/4049647799 wait complete. 2024-03-29T11:39:10.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:10.863+0000 7f2874cff640 1 librados: shutdown 2024-03-29T11:39:10.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:17.687 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.834+0000 7f37f0e74640 10 monclient: get_monmap_and_config 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.834+0000 7f37f0e74640 10 monclient: build_initial_monmap 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.834+0000 7f37f0e74640 10 monclient: monmap: 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:39:17.836052+0000 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:39:17.836052+0000 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:39:17.836 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:17.837 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.835+0000 7f37f0e74640 1 Processor -- start 2024-03-29T11:39:17.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.836+0000 7f37f0e74640 1 -- start start 2024-03-29T11:39:17.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.836+0000 7f37f0e74640 10 monclient: init 2024-03-29T11:39:17.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:17.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:17.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec1457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:17.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec13f650 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:17.841 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 10 monclient(hunting): picked mon.noname-a con 0x7f37ec0a9d70 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:17.841 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec0aaef0 con 0x7f37ec0a9d70 2024-03-29T11:39:17.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec145bd0 con 0x7f37ec1457f0 2024-03-29T11:39:17.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec145eb0 con 0x7f37ec13f650 2024-03-29T11:39:17.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:17.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.837+0000 7f37f0e74640 10 monclient(hunting): authenticate will time out at 1147.145167s 2024-03-29T11:39:17.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.840+0000 7f37ea59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f37ec1457f0 0x7f37ec13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:33629/0 (socket says 172.21.15.145:33629) 2024-03-29T11:39:17.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.840+0000 7f37ea59c640 1 -- 172.21.15.145:0/2901815509 learned_addr learned my addr 172.21.15.145:0/2901815509 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.841+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128298877 0 0) 0x7f37ec145bd0 con 0x7f37ec1457f0 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.841+0000 7f37e959a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.841+0000 7f37e959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.841+0000 7f37e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37d4003520 con 0x7f37ec1457f0 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040380414 0 0) 0x7f37ec0aaef0 con 0x7f37ec0a9d70 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:17.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:39:17.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec145bd0 con 0x7f37ec0a9d70 2024-03-29T11:39:17.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4190965674 0 0) 0x7f37d4003520 con 0x7f37ec1457f0 2024-03-29T11:39:17.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 >> v1:172.21.15.145:6790/0 conn(0x7f37ec13f650 legacy=0x7f37ec0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:17.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.842+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 >> v1:172.21.15.145:6789/0 conn(0x7f37ec0a9d70 legacy=0x7f37ec14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:17.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:17.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 1 monclient: found mon.noname-c 2024-03-29T11:39:17.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:17.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:39:17.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec146340 con 0x7f37ec1457f0 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 10 monclient: _finish_auth 0 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37f0e74640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37f0e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f37e4001a10 con 0x7f37ec1457f0 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 10 monclient: dump: 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:17.848 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37f0e74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37e4001c30 con 0x7f37ec1457f0 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37e959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37f0e74640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.845+0000 7f37f0e74640 4 set_mon_vals no callback set 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.846+0000 7f37e959a640 1 -- 172.21.15.145:0/2901815509 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f37e4001fc0 con 0x7f37ec1457f0 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.846+0000 7f37e959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.846+0000 7f37e959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.846+0000 7f37e959a640 10 monclient: dump: 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:17.849 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:17.850 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:17.850 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:17.850 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:17.850 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:17.850 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.846+0000 7f37f0e74640 10 monclient: shutdown 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.846+0000 7f37f0e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.846+0000 7f37f0e74640 1 -- 172.21.15.145:0/2901815509 >> v1:172.21.15.203:6789/0 conn(0x7f37ec1457f0 legacy=0x7f37ec13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.847+0000 7f37f0e74640 1 -- 172.21.15.145:0/2901815509 shutdown_connections 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.847+0000 7f37f0e74640 1 -- 172.21.15.145:0/2901815509 >> 172.21.15.145:0/2901815509 conn(0x7f37ec066a00 msgr2=0x7f37ec066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.847+0000 7f37f0e74640 1 -- 172.21.15.145:0/2901815509 shutdown_connections 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.847+0000 7f37f0e74640 1 -- 172.21.15.145:0/2901815509 wait complete. 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.848+0000 7f37f0e74640 10 monclient: build_initial_monmap 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.848+0000 7f37f0e74640 10 monclient: monmap: 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:39:17.852 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.848+0000 7f37f0e74640 1 librados: starting msgr at 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.848+0000 7f37f0e74640 1 librados: starting objecter 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.848+0000 7f37f0e74640 1 Processor -- start 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.848+0000 7f37f0e74640 1 -- start start 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.848+0000 7f37f0e74640 1 librados: setting wanted keys 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.848+0000 7f37f0e74640 1 librados: calling monclient init 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.848+0000 7f37f0e74640 10 monclient: init 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 10 monclient(hunting): picked mon.noname-c con 0x7f37ec0a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:17.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 10 monclient(hunting): picked mon.noname-a con 0x7f37ec13f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:17.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 10 monclient(hunting): picked mon.noname-b con 0x7f37ec1457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:17.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec06fb10 con 0x7f37ec13f650 2024-03-29T11:39:17.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec06e810 con 0x7f37ec1457f0 2024-03-29T11:39:17.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ec06fd80 con 0x7f37ec0a9d70 2024-03-29T11:39:17.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:17.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.849+0000 7f37f0e74640 10 monclient(hunting): authenticate will time out at 1147.157894s 2024-03-29T11:39:17.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.853+0000 7f37ead9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f37ec1457f0 0x7f37ec13fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:33689/0 (socket says 172.21.15.145:33689) 2024-03-29T11:39:17.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.854+0000 7f37ead9d640 1 -- 172.21.15.145:0/2827872618 learned_addr learned my addr 172.21.15.145:0/2827872618 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:17.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.854+0000 7f37daffd640 10 client.?.objecter ms_handle_connect 0x7f37ec1457f0 2024-03-29T11:39:17.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.854+0000 7f37daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287253244 0 0) 0x7f37ec06e810 con 0x7f37ec1457f0 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37cc004eb0 con 0x7f37ec1457f0 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 10 client.?.objecter ms_handle_connect 0x7f37ec13f650 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2032380715 0 0) 0x7f37cc004eb0 con 0x7f37ec1457f0 2024-03-29T11:39:17.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.855+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ec06e810 con 0x7f37ec1457f0 2024-03-29T11:39:17.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.856+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358906400 0 0) 0x7f37ec06fb10 con 0x7f37ec13f650 2024-03-29T11:39:17.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.856+0000 7f37daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:17.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.856+0000 7f37daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:17.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.856+0000 7f37daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:17.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:39:17.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37cc004eb0 con 0x7f37ec13f650 2024-03-29T11:39:17.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 10 client.?.objecter ms_handle_connect 0x7f37ec0a9d70 2024-03-29T11:39:17.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:17.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f37e0001990 con 0x7f37ec1457f0 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 10 monclient(hunting): dump: 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826419223 0 0) 0x7f37ec06fd80 con 0x7f37ec0a9d70 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.857+0000 7f37daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:17.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37ec06fb10 con 0x7f37ec0a9d70 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1462890155 0 0) 0x7f37cc004eb0 con 0x7f37ec13f650 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37ec06fd80 con 0x7f37ec13f650 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1132359491 0 0) 0x7f37ec06e810 con 0x7f37ec1457f0 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 >> v1:172.21.15.145:6790/0 conn(0x7f37ec0a9d70 legacy=0x7f37ec06e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 >> v1:172.21.15.145:6789/0 conn(0x7f37ec13f650 legacy=0x7f37ec0715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.858+0000 7f37daffd640 1 monclient: found mon.b 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37daffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ec06ea50 con 0x7f37ec1457f0 2024-03-29T11:39:17.863 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37daffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37f0e74640 5 monclient: authenticate success, global_id 4387 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37f0e74640 10 monclient: _renew_subs 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37f0e74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ec06eca0 con 0x7f37ec1457f0 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37f0e74640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37f0e74640 10 monclient: _renew_subs 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37f0e74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ec071ed0 con 0x7f37ec1457f0 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.859+0000 7f37f0e74640 1 librados: init done 2024-03-29T11:39:17.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.860+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37e0001af0 con 0x7f37ec1457f0 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37f0e74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37f0e74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37f0e74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ec142a90 con 0x7f37ec1457f0 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f37e0001d80 con 0x7f37ec1457f0 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37daffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37daffd640 10 monclient: dump: 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:17.865 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:17.866 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:17.866 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:17.866 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:17.866 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:17.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.861+0000 7f37e8d99640 4 set_mon_vals no callback set 2024-03-29T11:39:17.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.864+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f37e0003a60 con 0x7f37ec1457f0 2024-03-29T11:39:17.867 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.865+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f37e0084a10 con 0x7f37ec1457f0 2024-03-29T11:39:17.867 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.865+0000 7f37daffd640 10 client.4387.objecter ms_dispatch 0x7f37ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:17.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.865+0000 7f37daffd640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:17.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.865+0000 7f37daffd640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:17.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.865+0000 7f37ea59c640 10 monclient: get_auth_request con 0x7f37cc06ded0 auth_method 0 2024-03-29T11:39:17.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.865+0000 7f37daffd640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-29T11:39:17.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.865+0000 7f37daffd640 10 client.4387.objecter ms_handle_connect 0x7f37cc06ded0 2024-03-29T11:39:17.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.873+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37e0050120 con 0x7f37ec1457f0 2024-03-29T11:39:17.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.873+0000 7f37daffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:17.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:17.873+0000 7f37daffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:18.197 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.194+0000 7f37f0e74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:18.197 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.194+0000 7f37f0e74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:18.197 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.194+0000 7f37f0e74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:18.197 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.194+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37ec075f10 con 0x7f37ec1457f0 2024-03-29T11:39:18.202 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:39:18.203 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:18.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.200+0000 7f37daffd640 1 -- 172.21.15.145:0/2827872618 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f37e0052220 con 0x7f37ec1457f0 2024-03-29T11:39:18.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.200+0000 7f37daffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:18.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.200+0000 7f37daffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:18.206 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.204+0000 7f37f0e74640 10 librados: watch_flush enter 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 10 librados: watch_flush exit 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 10 client.4387.objecter successfully canceled tick 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 >> v1:172.21.15.145:6816/227372239 conn(0x7f37cc06ded0 legacy=0x7f37cc070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 10 monclient: shutdown 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 >> v1:172.21.15.203:6789/0 conn(0x7f37ec1457f0 legacy=0x7f37ec13fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 shutdown_connections 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 >> 172.21.15.145:0/2827872618 conn(0x7f37ec066a00 msgr2=0x7f37ec0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 shutdown_connections 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 1 -- 172.21.15.145:0/2827872618 wait complete. 2024-03-29T11:39:18.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:18.205+0000 7f37f0e74640 1 librados: shutdown 2024-03-29T11:39:18.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:25.031 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:25.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.170+0000 7fa129d1c640 10 monclient: get_monmap_and_config 2024-03-29T11:39:25.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.170+0000 7fa129d1c640 10 monclient: build_initial_monmap 2024-03-29T11:39:25.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.170+0000 7fa129d1c640 10 monclient: monmap: 2024-03-29T11:39:25.172 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:25.172 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:25.172 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:39:25.172057+0000 2024-03-29T11:39:25.172 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:39:25.172057+0000 2024-03-29T11:39:25.173 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:25.173 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:25.173 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:25.173 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:39:25.173 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:39:25.173 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:25.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.171+0000 7fa129d1c640 1 Processor -- start 2024-03-29T11:39:25.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.172+0000 7fa129d1c640 1 -- start start 2024-03-29T11:39:25.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.172+0000 7fa129d1c640 10 monclient: init 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.172+0000 7fa129d1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.172+0000 7fa129d1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa129d1c640 10 monclient(hunting): picked mon.noname-b con 0x7fa1241457f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa129d1c640 10 monclient(hunting): picked mon.noname-a con 0x7fa12413f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa129d1c640 10 monclient(hunting): picked mon.noname-c con 0x7fa1240a9d70 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1237fe640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fa1241457f0 0x7fa12413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:42914/0 (socket says 172.21.15.145:42914) 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1237fe640 1 -- 172.21.15.145:0/2122248616 learned_addr learned my addr 172.21.15.145:0/2122248616 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa129d1c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1240aaef0 con 0x7fa12413f650 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa129d1c640 1 -- 172.21.15.145:0/2122248616 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa124145bd0 con 0x7fa1240a9d70 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa129d1c640 1 -- 172.21.15.145:0/2122248616 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa124145eb0 con 0x7fa1241457f0 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa129d1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa129d1c640 10 monclient(hunting): authenticate will time out at 1154.481320s 2024-03-29T11:39:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1227fc640 1 -- 172.21.15.145:0/2122248616 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853153077 0 0) 0x7fa124145eb0 con 0x7fa1241457f0 2024-03-29T11:39:25.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:39:25.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:25.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:25.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1227fc640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:39:25.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1227fc640 1 -- 172.21.15.145:0/2122248616 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa110003520 con 0x7fa1241457f0 2024-03-29T11:39:25.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1227fc640 1 -- 172.21.15.145:0/2122248616 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1191731273 0 0) 0x7fa110003520 con 0x7fa1241457f0 2024-03-29T11:39:25.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1227fc640 1 -- 172.21.15.145:0/2122248616 >> v1:172.21.15.203:6789/0 conn(0x7fa1240a9d70 legacy=0x7fa12414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:39:25.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.173+0000 7fa1227fc640 1 -- 172.21.15.145:0/2122248616 >> v1:172.21.15.145:6789/0 conn(0x7fa12413f650 legacy=0x7fa1240b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 1 monclient: found mon.noname-b 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 1 -- 172.21.15.145:0/2122248616 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa124146340 con 0x7fa1241457f0 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 10 monclient: _finish_auth 0 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 1 -- 172.21.15.145:0/2122248616 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa114001690 con 0x7fa1241457f0 2024-03-29T11:39:25.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa1227fc640 10 monclient: dump: 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa129d1c640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:39:25.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.175+0000 7fa129d1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:25.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.176+0000 7fa1227fc640 1 -- 172.21.15.145:0/2122248616 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa114001860 con 0x7fa1241457f0 2024-03-29T11:39:25.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.176+0000 7fa1227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.176+0000 7fa129d1c640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.176+0000 7fa129d1c640 4 set_mon_vals no callback set 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.176+0000 7fa1227fc640 1 -- 172.21.15.145:0/2122248616 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa114001cb0 con 0x7fa1241457f0 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.176+0000 7fa1227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.176+0000 7fa1227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.176+0000 7fa1227fc640 10 monclient: dump: 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:25.180 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:25.181 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:25.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.177+0000 7fa129d1c640 10 monclient: shutdown 2024-03-29T11:39:25.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.177+0000 7fa129d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:25.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.177+0000 7fa129d1c640 1 -- 172.21.15.145:0/2122248616 >> v1:172.21.15.145:6790/0 conn(0x7fa1241457f0 legacy=0x7fa12413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:25.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.177+0000 7fa129d1c640 1 -- 172.21.15.145:0/2122248616 shutdown_connections 2024-03-29T11:39:25.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.177+0000 7fa129d1c640 1 -- 172.21.15.145:0/2122248616 >> 172.21.15.145:0/2122248616 conn(0x7fa124066a00 msgr2=0x7fa124066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:25.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.178+0000 7fa129d1c640 1 -- 172.21.15.145:0/2122248616 shutdown_connections 2024-03-29T11:39:25.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.179+0000 7fa129d1c640 1 -- 172.21.15.145:0/2122248616 wait complete. 2024-03-29T11:39:25.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.180+0000 7fa129d1c640 10 monclient: build_initial_monmap 2024-03-29T11:39:25.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.180+0000 7fa129d1c640 10 monclient: monmap: 2024-03-29T11:39:25.182 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:25.182 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:25.182 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:39:25.182 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:39:25.182 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:25.183 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:25.183 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:25.183 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:39:25.183 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:39:25.183 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:25.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.180+0000 7fa129d1c640 1 librados: starting msgr at 2024-03-29T11:39:25.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.180+0000 7fa129d1c640 1 librados: starting objecter 2024-03-29T11:39:25.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.181+0000 7fa129d1c640 1 Processor -- start 2024-03-29T11:39:25.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.181+0000 7fa129d1c640 1 -- start start 2024-03-29T11:39:25.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.181+0000 7fa129d1c640 1 librados: setting wanted keys 2024-03-29T11:39:25.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.181+0000 7fa129d1c640 1 librados: calling monclient init 2024-03-29T11:39:25.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.181+0000 7fa129d1c640 10 monclient: init 2024-03-29T11:39:25.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa129d1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa129d1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa129d1c640 10 monclient(hunting): picked mon.noname-b con 0x7fa1240a9d70 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa129d1c640 10 monclient(hunting): picked mon.noname-a con 0x7fa12413f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa129d1c640 10 monclient(hunting): picked mon.noname-c con 0x7fa1241457f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa129d1c640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12413c5d0 con 0x7fa12413f650 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa129d1c640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12413c280 con 0x7fa1240a9d70 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa123fff640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fa1241457f0 0x7fa1241d7210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:43566/0 (socket says 172.21.15.145:43566) 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa123fff640 1 -- 172.21.15.145:0/1534570621 learned_addr learned my addr 172.21.15.145:0/1534570621 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa11bfff640 10 client.?.objecter ms_handle_connect 0x7fa1241457f0 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.182+0000 7fa11bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12413ac90 con 0x7fa1241457f0 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa129d1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa129d1c640 10 monclient(hunting): authenticate will time out at 1154.491258s 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2823860107 0 0) 0x7fa12413ac90 con 0x7fa1241457f0 2024-03-29T11:39:25.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa11bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa11bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa11bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa11bfff640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa104004eb0 con 0x7fa1241457f0 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa11bfff640 10 client.?.objecter ms_handle_connect 0x7fa1240a9d70 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.183+0000 7fa11bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3355367980 0 0) 0x7fa104004eb0 con 0x7fa1241457f0 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa12413ac90 con 0x7fa1241457f0 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa11c001c60 con 0x7fa1241457f0 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 10 monclient(hunting): dump: 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:25.188 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2207405351 0 0) 0x7fa12413ac90 con 0x7fa1241457f0 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 >> v1:172.21.15.203:6789/0 conn(0x7fa1240a9d70 legacy=0x7fa12413a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:25.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.184+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 >> v1:172.21.15.145:6789/0 conn(0x7fa12413f650 legacy=0x7fa124135cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:25.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa11bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:25.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa11bfff640 1 monclient: found mon.c 2024-03-29T11:39:25.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa11bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:25.192 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa11bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12413ae20 con 0x7fa1241457f0 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa11bfff640 10 monclient: _finish_auth 0 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa11bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa11bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa129d1c640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa129d1c640 10 monclient: _renew_subs 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa129d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa124139800 con 0x7fa1241457f0 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa129d1c640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa129d1c640 10 monclient: _renew_subs 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa129d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1241365a0 con 0x7fa1241457f0 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.186+0000 7fa129d1c640 1 librados: init done 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.187+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa11c001e30 con 0x7fa1241457f0 2024-03-29T11:39:25.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.187+0000 7fa11bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.187+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa11c003c20 con 0x7fa1241457f0 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.187+0000 7fa11bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.187+0000 7fa11bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.187+0000 7fa11bfff640 10 monclient: dump: 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:25.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.187+0000 7fa121ffb640 4 set_mon_vals no callback set 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fa11c003d80 con 0x7fa1241457f0 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa129d1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa129d1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa129d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa108005080 con 0x7fa1241457f0 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fa11c084d60 con 0x7fa1241457f0 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa11bfff640 10 client.4388.objecter ms_dispatch 0x7fa124000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa11bfff640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa11bfff640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa1237fe640 10 monclient: get_auth_request con 0x7fa10406dca0 auth_method 0 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.189+0000 7fa11bfff640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T11:39:25.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.190+0000 7fa11bfff640 10 client.4388.objecter ms_handle_connect 0x7fa10406dca0 2024-03-29T11:39:25.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.196+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa11c050640 con 0x7fa1241457f0 2024-03-29T11:39:25.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.196+0000 7fa11bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:25.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.196+0000 7fa11bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:25.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.500+0000 7fa129d1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:25.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.500+0000 7fa129d1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:25.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.500+0000 7fa129d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:25.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.500+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa108005330 con 0x7fa1241457f0 2024-03-29T11:39:25.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.501+0000 7fa11bfff640 1 -- 172.21.15.145:0/1534570621 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fa11c052570 con 0x7fa1241457f0 2024-03-29T11:39:25.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.501+0000 7fa11bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:25.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.501+0000 7fa11bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:25.503 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:39:25.503 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:25.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.504+0000 7fa129d1c640 10 librados: watch_flush enter 2024-03-29T11:39:25.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.504+0000 7fa129d1c640 10 librados: watch_flush exit 2024-03-29T11:39:25.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.504+0000 7fa129d1c640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T11:39:25.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.504+0000 7fa129d1c640 10 client.4388.objecter successfully canceled tick 2024-03-29T11:39:25.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.505+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 >> v1:172.21.15.145:6816/227372239 conn(0x7fa10406dca0 legacy=0x7fa104070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:25.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.505+0000 7fa129d1c640 10 monclient: shutdown 2024-03-29T11:39:25.506 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.505+0000 7fa129d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:25.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.505+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 >> v1:172.21.15.145:6790/0 conn(0x7fa1241457f0 legacy=0x7fa1241d7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:25.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.505+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 shutdown_connections 2024-03-29T11:39:25.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.505+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 >> 172.21.15.145:0/1534570621 conn(0x7fa124066a00 msgr2=0x7fa124066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:25.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.506+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 shutdown_connections 2024-03-29T11:39:25.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.506+0000 7fa129d1c640 1 -- 172.21.15.145:0/1534570621 wait complete. 2024-03-29T11:39:25.507 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:25.506+0000 7fa129d1c640 1 librados: shutdown 2024-03-29T11:39:25.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:32.327 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:32.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.476+0000 7f2b023ef640 10 monclient: get_monmap_and_config 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.476+0000 7f2b023ef640 10 monclient: build_initial_monmap 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.476+0000 7f2b023ef640 10 monclient: monmap: 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:39:32.478140+0000 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:39:32.478140+0000 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:39:32.479 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:32.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.478+0000 7f2b023ef640 1 Processor -- start 2024-03-29T11:39:32.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.478+0000 7f2b023ef640 1 -- start start 2024-03-29T11:39:32.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.478+0000 7f2b023ef640 10 monclient: init 2024-03-29T11:39:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 10 monclient(hunting): picked mon.noname-b con 0x7f2af40389e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 10 monclient(hunting): picked mon.noname-c con 0x7f2af4038dc0 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 10 monclient(hunting): picked mon.noname-a con 0x7f2af4030660 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af40c54c0 con 0x7f2af4030660 2024-03-29T11:39:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af4036430 con 0x7f2af4038dc0 2024-03-29T11:39:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af4039840 con 0x7f2af40389e0 2024-03-29T11:39:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b023ef640 10 monclient(hunting): authenticate will time out at 1161.787581s 2024-03-29T11:39:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b00bec640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f2af4038dc0 0x7f2af40cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:58792/0 (socket says 172.21.15.145:58792) 2024-03-29T11:39:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.479+0000 7f2b00bec640 1 -- 172.21.15.145:0/3612861446 learned_addr learned my addr 172.21.15.145:0/3612861446 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.480+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1844294986 0 0) 0x7f2af4036430 con 0x7f2af4038dc0 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.480+0000 7f2af3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.480+0000 7f2af3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.480+0000 7f2af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.480+0000 7f2af3fff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.480+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae8003520 con 0x7f2af4038dc0 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139585114 0 0) 0x7f2af4039840 con 0x7f2af40389e0 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae8003840 con 0x7f2af40389e0 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 584300498 0 0) 0x7f2ae8003520 con 0x7f2af4038dc0 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 >> v1:172.21.15.145:6790/0 conn(0x7f2af40389e0 legacy=0x7f2af4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 >> v1:172.21.15.145:6789/0 conn(0x7f2af4030660 legacy=0x7f2af40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 monclient: found mon.noname-c 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af4035690 con 0x7f2af4038dc0 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2af8001670 con 0x7f2af4038dc0 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: dump: 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:32.485 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2b023ef640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2b023ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2af8001850 con 0x7f2af4038dc0 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2b023ef640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:32.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2b023ef640 4 set_mon_vals no callback set 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 1 -- 172.21.15.145:0/3612861446 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2af8001c60 con 0x7f2af4038dc0 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.481+0000 7f2af3fff640 10 monclient: dump: 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:32.487 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.482+0000 7f2b023ef640 10 monclient: shutdown 2024-03-29T11:39:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.482+0000 7f2b023ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.482+0000 7f2b023ef640 1 -- 172.21.15.145:0/3612861446 >> v1:172.21.15.203:6789/0 conn(0x7f2af4038dc0 legacy=0x7f2af40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.482+0000 7f2b023ef640 1 -- 172.21.15.145:0/3612861446 shutdown_connections 2024-03-29T11:39:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.482+0000 7f2b023ef640 1 -- 172.21.15.145:0/3612861446 >> 172.21.15.145:0/3612861446 conn(0x7f2af4005c00 msgr2=0x7f2af4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.483+0000 7f2b023ef640 1 -- 172.21.15.145:0/3612861446 shutdown_connections 2024-03-29T11:39:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.483+0000 7f2b023ef640 1 -- 172.21.15.145:0/3612861446 wait complete. 2024-03-29T11:39:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.484+0000 7f2b023ef640 10 monclient: build_initial_monmap 2024-03-29T11:39:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.484+0000 7f2b023ef640 10 monclient: monmap: 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.485+0000 7f2b023ef640 1 librados: starting msgr at 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.485+0000 7f2b023ef640 1 librados: starting objecter 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.485+0000 7f2b023ef640 1 Processor -- start 2024-03-29T11:39:32.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.485+0000 7f2b023ef640 1 -- start start 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.485+0000 7f2b023ef640 1 librados: setting wanted keys 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.485+0000 7f2b023ef640 1 librados: calling monclient init 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.485+0000 7f2b023ef640 10 monclient: init 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.486+0000 7f2b023ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.486+0000 7f2b023ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.486+0000 7f2b023ef640 10 monclient(hunting): picked mon.noname-b con 0x7f2af4030660 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.486+0000 7f2b023ef640 10 monclient(hunting): picked mon.noname-a con 0x7f2af40389e0 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.486+0000 7f2b023ef640 10 monclient(hunting): picked mon.noname-c con 0x7f2af40c7500 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.486+0000 7f2b023ef640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af402d8b0 con 0x7f2af40389e0 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.486+0000 7f2b023ef640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af40c7eb0 con 0x7f2af4030660 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.486+0000 7f2b023ef640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af40c7990 con 0x7f2af40c7500 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2b013ed640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f2af4030660 0x7f2af40bf210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:59036/0 (socket says 172.21.15.145:59036) 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2b013ed640 1 -- 172.21.15.145:0/799656617 learned_addr learned my addr 172.21.15.145:0/799656617 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2b023ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2b023ef640 10 monclient(hunting): authenticate will time out at 1161.795098s 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 client.?.objecter ms_handle_connect 0x7f2af4030660 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 client.?.objecter ms_handle_connect 0x7f2af40c7500 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 client.?.objecter ms_handle_connect 0x7f2af40389e0 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289203007 0 0) 0x7f2af40c7eb0 con 0x7f2af4030660 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.487+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae0004eb0 con 0x7f2af4030660 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 100241994 0 0) 0x7f2af402d8b0 con 0x7f2af40389e0 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af40c7eb0 con 0x7f2af40389e0 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 120627993 0 0) 0x7f2ae0004eb0 con 0x7f2af4030660 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2af402d8b0 con 0x7f2af4030660 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2aec001a10 con 0x7f2af4030660 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:32.497 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15864705 0 0) 0x7f2af40c7990 con 0x7f2af40c7500 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae0004eb0 con 0x7f2af40c7500 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2816379515 0 0) 0x7f2af402d8b0 con 0x7f2af4030660 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 >> v1:172.21.15.145:6790/0 conn(0x7f2af40c7500 legacy=0x7f2af41680f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 >> v1:172.21.15.145:6789/0 conn(0x7f2af40389e0 legacy=0x7f2af415c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 monclient: found mon.b 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af40c80c0 con 0x7f2af4030660 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.488+0000 7f2af1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2b023ef640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2b023ef640 10 monclient: _renew_subs 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2b023ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af40c6c90 con 0x7f2af4030660 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2aec001c50 con 0x7f2af4030660 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2af1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2aec003a60 con 0x7f2af4030660 2024-03-29T11:39:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2af1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2af1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2af1ffb640 10 monclient: dump: 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.489+0000 7f2af2ffd640 4 set_mon_vals no callback set 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.490+0000 7f2b023ef640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:32.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.490+0000 7f2b023ef640 10 monclient: _renew_subs 2024-03-29T11:39:32.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.490+0000 7f2b023ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:32.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.490+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af40bfb00 con 0x7f2af4030660 2024-03-29T11:39:32.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.491+0000 7f2b023ef640 1 librados: init done 2024-03-29T11:39:32.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.491+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f2aec003bc0 con 0x7f2af4030660 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.492+0000 7f2b00bec640 10 monclient: get_auth_request con 0x7f2ae006ddf0 auth_method 0 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.492+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f2aec084b10 con 0x7f2af4030660 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.492+0000 7f2af1ffb640 10 client.4396.objecter ms_dispatch 0x7f2afc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.492+0000 7f2af1ffb640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.492+0000 7f2af1ffb640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.492+0000 7f2af1ffb640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.493+0000 7f2af1ffb640 10 client.4396.objecter ms_handle_connect 0x7f2ae006ddf0 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.493+0000 7f2b023ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.493+0000 7f2b023ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.493+0000 7f2b023ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:32.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.493+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2af4007b70 con 0x7f2af4030660 2024-03-29T11:39:32.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.500+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2aec0503f0 con 0x7f2af4030660 2024-03-29T11:39:32.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.500+0000 7f2af1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:32.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.500+0000 7f2af1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:32.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.878+0000 7f2b023ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:32.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.878+0000 7f2b023ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:32.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.878+0000 7f2b023ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:39:32.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.878+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2af40c7b20 con 0x7f2af4030660 2024-03-29T11:39:32.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.879+0000 7f2af1ffb640 1 -- 172.21.15.145:0/799656617 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f2aec052320 con 0x7f2af4030660 2024-03-29T11:39:32.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.879+0000 7f2af1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:32.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.879+0000 7f2af1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:32.882 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:39:32.882 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.883+0000 7f2b023ef640 10 librados: watch_flush enter 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.883+0000 7f2b023ef640 10 librados: watch_flush exit 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.883+0000 7f2b023ef640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.883+0000 7f2b023ef640 10 client.4396.objecter successfully canceled tick 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.884+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 >> v1:172.21.15.145:6816/227372239 conn(0x7f2ae006ddf0 legacy=0x7f2ae00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.884+0000 7f2b023ef640 10 monclient: shutdown 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.884+0000 7f2b023ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.884+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 >> v1:172.21.15.203:6789/0 conn(0x7f2af4030660 legacy=0x7f2af40bf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.884+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 shutdown_connections 2024-03-29T11:39:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.884+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 >> 172.21.15.145:0/799656617 conn(0x7f2af4005c00 msgr2=0x7f2af40c4770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:32.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.884+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 shutdown_connections 2024-03-29T11:39:32.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.884+0000 7f2b023ef640 1 -- 172.21.15.145:0/799656617 wait complete. 2024-03-29T11:39:32.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:32.884+0000 7f2b023ef640 1 librados: shutdown 2024-03-29T11:39:32.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:39.707 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:39.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.840+0000 7f6b435c4640 10 monclient: get_monmap_and_config 2024-03-29T11:39:39.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.840+0000 7f6b435c4640 10 monclient: build_initial_monmap 2024-03-29T11:39:39.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.840+0000 7f6b435c4640 10 monclient: monmap: 2024-03-29T11:39:39.842 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:39.843 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:39.843 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:39:39.841693+0000 2024-03-29T11:39:39.843 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:39:39.841693+0000 2024-03-29T11:39:39.843 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:39.843 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:39.843 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:39.843 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:39:39.843 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:39:39.843 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:39.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.842+0000 7f6b435c4640 1 Processor -- start 2024-03-29T11:39:39.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.842+0000 7f6b435c4640 1 -- start start 2024-03-29T11:39:39.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.842+0000 7f6b435c4640 10 monclient: init 2024-03-29T11:39:39.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:39.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:39:39.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 10 monclient(hunting): picked mon.noname-c con 0x7f6b3c0ae250 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:39.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 10 monclient(hunting): picked mon.noname-a con 0x7f6b3c13f970 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:39.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 10 monclient(hunting): picked mon.noname-b con 0x7f6b3c142800 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:39.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c0afe00 con 0x7f6b3c13f970 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c142c10 con 0x7f6b3c0ae250 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c0afb20 con 0x7f6b3c142800 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b435c4640 10 monclient(hunting): authenticate will time out at 1169.151421s 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b40b5f640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f6b3c13f970 0x7f6b3c1420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:43002/0 (socket says 172.21.15.145:43002) 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b41b61640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f6b3c142800 0x7f6b3c14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:35818/0 (socket says 172.21.15.145:35818) 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.843+0000 7f6b40b5f640 1 -- 172.21.15.145:0/1815118157 learned_addr learned my addr 172.21.15.145:0/1815118157 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245650496 0 0) 0x7f6b3c0afb20 con 0x7f6b3c142800 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:39.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b28003520 con 0x7f6b3c142800 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464791595 0 0) 0x7f6b3c0afe00 con 0x7f6b3c13f970 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 10 monclient(hunting): _negotiate 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b3c0afb20 con 0x7f6b3c13f970 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3946277365 0 0) 0x7f6b28003520 con 0x7f6b3c142800 2024-03-29T11:39:39.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.844+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 >> v1:172.21.15.203:6789/0 conn(0x7f6b3c0ae250 legacy=0x7f6b3c13f260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:39.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 >> v1:172.21.15.145:6789/0 conn(0x7f6b3c13f970 legacy=0x7f6b3c1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 10 monclient: _finish_hunting 0 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 1 monclient: found mon.noname-b 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c0aab80 con 0x7f6b3c142800 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 10 monclient: _finish_auth 0 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 10 monclient: _check_auth_tickets 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f6b34001a10 con 0x7f6b3c142800 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b435c4640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.846+0000 7f6b435c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b33fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:39.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b33fff640 10 monclient: dump: 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b435c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b34001b70 con 0x7f6b3c142800 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:39.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b33fff640 1 -- 172.21.15.145:0/1815118157 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f6b34001fc0 con 0x7f6b3c142800 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b435c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b435c4640 4 set_mon_vals no callback set 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.847+0000 7f6b33fff640 10 monclient: dump: 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:39.851 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:39.852 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:39.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.848+0000 7f6b435c4640 10 monclient: shutdown 2024-03-29T11:39:39.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.848+0000 7f6b435c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:39.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.848+0000 7f6b435c4640 1 -- 172.21.15.145:0/1815118157 >> v1:172.21.15.145:6790/0 conn(0x7f6b3c142800 legacy=0x7f6b3c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:39.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.849+0000 7f6b435c4640 1 -- 172.21.15.145:0/1815118157 shutdown_connections 2024-03-29T11:39:39.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.849+0000 7f6b435c4640 1 -- 172.21.15.145:0/1815118157 >> 172.21.15.145:0/1815118157 conn(0x7f6b3c0b35e0 msgr2=0x7f6b3c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:39.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.849+0000 7f6b435c4640 1 -- 172.21.15.145:0/1815118157 shutdown_connections 2024-03-29T11:39:39.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.849+0000 7f6b435c4640 1 -- 172.21.15.145:0/1815118157 wait complete. 2024-03-29T11:39:39.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.850+0000 7f6b435c4640 10 monclient: build_initial_monmap 2024-03-29T11:39:39.854 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.850+0000 7f6b435c4640 10 monclient: monmap: 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.850+0000 7f6b435c4640 1 librados: starting msgr at 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.850+0000 7f6b435c4640 1 librados: starting objecter 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.850+0000 7f6b435c4640 1 Processor -- start 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.850+0000 7f6b435c4640 1 -- start start 2024-03-29T11:39:39.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.850+0000 7f6b435c4640 1 librados: setting wanted keys 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.850+0000 7f6b435c4640 1 librados: calling monclient init 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.850+0000 7f6b435c4640 10 monclient: init 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 10 monclient(hunting): picked mon.noname-c con 0x7f6b3c0ae250 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 10 monclient(hunting): picked mon.noname-a con 0x7f6b3c13f970 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 10 monclient(hunting): picked mon.noname-b con 0x7f6b3c142800 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c144060 con 0x7f6b3c13f970 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c13bd60 con 0x7f6b3c142800 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b3c13c280 con 0x7f6b3c0ae250 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.851+0000 7f6b435c4640 10 monclient(hunting): authenticate will time out at 1169.159798s 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.852+0000 7f6b40b5f640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f6b3c13f970 0x7f6b3c135a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:43224/0 (socket says 172.21.15.145:43224) 2024-03-29T11:39:39.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.852+0000 7f6b40b5f640 1 -- 172.21.15.145:0/3334618310 learned_addr learned my addr 172.21.15.145:0/3334618310 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:39.858 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 client.?.objecter ms_handle_connect 0x7f6b3c13f970 2024-03-29T11:39:39.858 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:39.858 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 client.?.objecter ms_handle_connect 0x7f6b3c0ae250 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808850034 0 0) 0x7f6b3c13c280 con 0x7f6b3c0ae250 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b20004eb0 con 0x7f6b3c0ae250 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3870521865 0 0) 0x7f6b20004eb0 con 0x7f6b3c0ae250 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b3c13c280 con 0x7f6b3c0ae250 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f6b380017a0 con 0x7f6b3c0ae250 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 monclient(hunting): dump: 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:39.859 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1848457479 0 0) 0x7f6b3c13c280 con 0x7f6b3c0ae250 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.853+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 >> v1:172.21.15.203:6789/0 conn(0x7f6b3c142800 legacy=0x7f6b3c1e2960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 >> v1:172.21.15.145:6789/0 conn(0x7f6b3c13f970 legacy=0x7f6b3c135a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b31ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:39.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b31ffb640 1 monclient: found mon.c 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b31ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b31ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c0ac9e0 con 0x7f6b3c0ae250 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b31ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b31ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b435c4640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b435c4640 10 monclient: _renew_subs 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b435c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b3c1dacd0 con 0x7f6b3c0ae250 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b435c4640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b435c4640 10 monclient: _renew_subs 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b435c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b3c13b270 con 0x7f6b3c0ae250 2024-03-29T11:39:39.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.855+0000 7f6b435c4640 1 librados: init done 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.856+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b38001900 con 0x7f6b3c0ae250 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.856+0000 7f6b31ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.856+0000 7f6b337fe640 4 set_mon_vals no callback set 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.857+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f6b38003830 con 0x7f6b3c0ae250 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.857+0000 7f6b31ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.857+0000 7f6b31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.857+0000 7f6b31ffb640 10 monclient: dump: 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:39.862 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:39.863 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:39.863 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.858+0000 7f6b435c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.858+0000 7f6b435c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.858+0000 7f6b435c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.858+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1c005080 con 0x7f6b3c0ae250 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.858+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f6b38019800 con 0x7f6b3c0ae250 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.859+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f6b380837d0 con 0x7f6b3c0ae250 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.859+0000 7f6b31ffb640 10 client.4400.objecter ms_dispatch 0x7f6b3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.859+0000 7f6b31ffb640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.859+0000 7f6b31ffb640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:39.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.859+0000 7f6b40b5f640 10 monclient: get_auth_request con 0x7f6b2006db60 auth_method 0 2024-03-29T11:39:39.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.859+0000 7f6b31ffb640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T11:39:39.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.859+0000 7f6b31ffb640 10 client.4400.objecter ms_handle_connect 0x7f6b2006db60 2024-03-29T11:39:39.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.866+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b3804f0b0 con 0x7f6b3c0ae250 2024-03-29T11:39:39.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.866+0000 7f6b31ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:39.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:39.866+0000 7f6b31ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:40.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.178+0000 7f6b435c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:40.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.179+0000 7f6b435c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:40.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.179+0000 7f6b435c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:40.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.179+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b1c005330 con 0x7f6b3c0ae250 2024-03-29T11:39:40.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.180+0000 7f6b31ffb640 1 -- 172.21.15.145:0/3334618310 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f6b38050fe0 con 0x7f6b3c0ae250 2024-03-29T11:39:40.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.180+0000 7f6b31ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:40.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.180+0000 7f6b31ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:40.182 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:39:40.182 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:40.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.183+0000 7f6b435c4640 10 librados: watch_flush enter 2024-03-29T11:39:40.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.183+0000 7f6b435c4640 10 librados: watch_flush exit 2024-03-29T11:39:40.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.183+0000 7f6b435c4640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T11:39:40.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.183+0000 7f6b435c4640 10 client.4400.objecter successfully canceled tick 2024-03-29T11:39:40.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.184+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 >> v1:172.21.15.145:6816/227372239 conn(0x7f6b2006db60 legacy=0x7f6b20070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:40.186 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.184+0000 7f6b435c4640 10 monclient: shutdown 2024-03-29T11:39:40.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.184+0000 7f6b435c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:40.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.184+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 >> v1:172.21.15.145:6790/0 conn(0x7f6b3c0ae250 legacy=0x7f6b3c1da5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:40.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.184+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 shutdown_connections 2024-03-29T11:39:40.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.184+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 >> 172.21.15.145:0/3334618310 conn(0x7f6b3c0b35e0 msgr2=0x7f6b3c0b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:40.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.184+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 shutdown_connections 2024-03-29T11:39:40.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.184+0000 7f6b435c4640 1 -- 172.21.15.145:0/3334618310 wait complete. 2024-03-29T11:39:40.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:40.184+0000 7f6b435c4640 1 librados: shutdown 2024-03-29T11:39:40.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:47.007 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:47.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.155+0000 7f7a08a86640 10 monclient: get_monmap_and_config 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.155+0000 7f7a08a86640 10 monclient: build_initial_monmap 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.155+0000 7f7a08a86640 10 monclient: monmap: 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:39:47.156446+0000 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:39:47.156446+0000 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:47.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.156+0000 7f7a08a86640 1 Processor -- start 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.156+0000 7f7a08a86640 1 -- start start 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.156+0000 7f7a08a86640 10 monclient: init 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 10 monclient(hunting): picked mon.noname-b con 0x7f7a040a9ac0 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 10 monclient(hunting): picked mon.noname-a con 0x7f7a0413ee90 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 10 monclient(hunting): picked mon.noname-c con 0x7f7a040af900 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a040afd10 con 0x7f7a0413ee90 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a040af6f0 con 0x7f7a040af900 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a040aac80 con 0x7f7a040a9ac0 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:47.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.157+0000 7f7a08a86640 10 monclient(hunting): authenticate will time out at 1176.465482s 2024-03-29T11:39:47.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.159+0000 7f7a0259c640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f7a040a9ac0 0x7f7a040adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:34954/0 (socket says 172.21.15.145:34954) 2024-03-29T11:39:47.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.159+0000 7f7a0259c640 1 -- 172.21.15.145:0/3246195088 learned_addr learned my addr 172.21.15.145:0/3246195088 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:47.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3423859444 0 0) 0x7f7a040af6f0 con 0x7f7a040af900 2024-03-29T11:39:47.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:47.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:47.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:47.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:39:47.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79ec003520 con 0x7f7a040af900 2024-03-29T11:39:47.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173101237 0 0) 0x7f7a040aac80 con 0x7f7a040a9ac0 2024-03-29T11:39:47.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 10 monclient(hunting): _negotiate 2024-03-29T11:39:47.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:47.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:47.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 10 monclient(hunting): my global_id is 4403 2024-03-29T11:39:47.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a040af6f0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 798860624 0 0) 0x7f7a040af6f0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 >> v1:172.21.15.203:6789/0 conn(0x7f7a040af900 legacy=0x7f7a04149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:47.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.161+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 >> v1:172.21.15.145:6789/0 conn(0x7f7a0413ee90 legacy=0x7f7a0413b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:47.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 10 monclient: _finish_hunting 0 2024-03-29T11:39:47.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 1 monclient: found mon.noname-b 2024-03-29T11:39:47.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:47.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:39:47.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a040aaf60 con 0x7f7a040a9ac0 2024-03-29T11:39:47.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 10 monclient: _finish_auth 0 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 10 monclient: _check_auth_tickets 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f79f8001670 con 0x7f7a040a9ac0 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.162+0000 7f7a0159a640 10 monclient: dump: 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:47.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a08a86640 5 monclient: authenticate success, global_id 4403 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a08a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79f80017d0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a0159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a0159a640 1 -- 172.21.15.145:0/3246195088 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f79f8001be0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a0159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a0159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a0159a640 10 monclient: dump: 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:47.168 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:47.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a08a86640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:47.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.163+0000 7f7a08a86640 4 set_mon_vals no callback set 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.165+0000 7f7a08a86640 10 monclient: shutdown 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.165+0000 7f7a08a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.165+0000 7f7a08a86640 1 -- 172.21.15.145:0/3246195088 >> v1:172.21.15.145:6790/0 conn(0x7f7a040a9ac0 legacy=0x7f7a040adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.166+0000 7f7a08a86640 1 -- 172.21.15.145:0/3246195088 shutdown_connections 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.166+0000 7f7a08a86640 1 -- 172.21.15.145:0/3246195088 >> 172.21.15.145:0/3246195088 conn(0x7f7a040b35e0 msgr2=0x7f7a040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.167+0000 7f7a08a86640 1 -- 172.21.15.145:0/3246195088 shutdown_connections 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.167+0000 7f7a08a86640 1 -- 172.21.15.145:0/3246195088 wait complete. 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.168+0000 7f7a08a86640 10 monclient: build_initial_monmap 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.168+0000 7f7a08a86640 10 monclient: monmap: 2024-03-29T11:39:47.170 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:47.171 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:47.171 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:39:47.171 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:39:47.171 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:47.171 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:47.171 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:47.171 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:39:47.171 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:39:47.171 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:47.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.169+0000 7f7a08a86640 1 librados: starting msgr at 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.169+0000 7f7a08a86640 1 librados: starting objecter 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.169+0000 7f7a08a86640 1 Processor -- start 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.169+0000 7f7a08a86640 1 -- start start 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.169+0000 7f7a08a86640 1 librados: setting wanted keys 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.169+0000 7f7a08a86640 1 librados: calling monclient init 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.169+0000 7f7a08a86640 10 monclient: init 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.170+0000 7f7a08a86640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.170+0000 7f7a08a86640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.170+0000 7f7a08a86640 10 monclient(hunting): picked mon.noname-c con 0x7f7a040a9ac0 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.170+0000 7f7a08a86640 10 monclient(hunting): picked mon.noname-a con 0x7f7a040af900 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.170+0000 7f7a08a86640 10 monclient(hunting): picked mon.noname-b con 0x7f7a0413ee90 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.170+0000 7f7a08a86640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a041c6fd0 con 0x7f7a040af900 2024-03-29T11:39:47.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.170+0000 7f7a08a86640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a041d6c80 con 0x7f7a0413ee90 2024-03-29T11:39:47.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.172+0000 7f7a08a86640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a041d6910 con 0x7f7a040a9ac0 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.172+0000 7f7a08a86640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.172+0000 7f7a08a86640 10 monclient(hunting): authenticate will time out at 1176.480638s 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.172+0000 7f7a0259c640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f7a040a9ac0 0x7f7a041d6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:35460/0 (socket says 172.21.15.145:35460) 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.172+0000 7f7a0259c640 1 -- 172.21.15.145:0/2334179402 learned_addr learned my addr 172.21.15.145:0/2334179402 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.172+0000 7f79f2ffd640 10 client.?.objecter ms_handle_connect 0x7f7a040a9ac0 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.172+0000 7f79f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558736566 0 0) 0x7f7a041d6910 con 0x7f7a040a9ac0 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79e4004eb0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3632566050 0 0) 0x7f79e4004eb0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a041d6910 con 0x7f7a040a9ac0 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f79f8001be0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.176 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 monclient(hunting): dump: 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 client.?.objecter ms_handle_connect 0x7f7a0413ee90 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:47.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2837643086 0 0) 0x7f7a041d6910 con 0x7f7a040a9ac0 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 >> v1:172.21.15.203:6789/0 conn(0x7f7a0413ee90 legacy=0x7f7a041c4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.173+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 >> v1:172.21.15.145:6789/0 conn(0x7f7a040af900 legacy=0x7f7a041c8830 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f79f2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f79f2ffd640 1 monclient: found mon.c 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f79f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f79f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a041395c0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f79f2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f79f2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:39:47.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f79f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:47.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f7a08a86640 5 monclient: authenticate success, global_id 4406 2024-03-29T11:39:47.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f7a08a86640 10 monclient: _renew_subs 2024-03-29T11:39:47.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f7a08a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:47.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.175+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a04139890 con 0x7f7a040a9ac0 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.176+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79f8004b10 con 0x7f7a040a9ac0 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.176+0000 7f79f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.176+0000 7f7a08a86640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.176+0000 7f7a08a86640 10 monclient: _renew_subs 2024-03-29T11:39:47.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.176+0000 7f7a00d99640 4 set_mon_vals no callback set 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.176+0000 7f7a08a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.176+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a041c9120 con 0x7f7a040a9ac0 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.176+0000 7f7a08a86640 1 librados: init done 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.177+0000 7f7a08a86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.177+0000 7f7a08a86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.177+0000 7f7a08a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.177+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a041d6ec0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.178+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f79f8004f60 con 0x7f7a040a9ac0 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.178+0000 7f79f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.178+0000 7f79f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.178+0000 7f79f2ffd640 10 monclient: dump: 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:47.182 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.178+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f79f8005140 con 0x7f7a040a9ac0 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.179+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f79f8054cf0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.179+0000 7f79f2ffd640 10 client.4406.objecter ms_dispatch 0x7f7a04000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.179+0000 7f79f2ffd640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.179+0000 7f79f2ffd640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.179+0000 7f79f2ffd640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.179+0000 7f7a01d9b640 10 monclient: get_auth_request con 0x7f79e406dcf0 auth_method 0 2024-03-29T11:39:47.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.179+0000 7f79f2ffd640 10 client.4406.objecter ms_handle_connect 0x7f79e406dcf0 2024-03-29T11:39:47.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.186+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a041d6ec0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.186+0000 7f79f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:47.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.186+0000 7f79f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:47.511 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.509+0000 7f7a08a86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:47.511 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.509+0000 7f7a08a86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:47.511 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.509+0000 7f7a08a86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:47.511 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.509+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a041d6ec0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.512 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.510+0000 7f79f2ffd640 1 -- 172.21.15.145:0/2334179402 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f7a041d6ec0 con 0x7f7a040a9ac0 2024-03-29T11:39:47.512 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.510+0000 7f79f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:47.512 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.510+0000 7f79f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:47.513 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:39:47.513 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:47.515 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.514+0000 7f7a08a86640 10 librados: watch_flush enter 2024-03-29T11:39:47.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.514+0000 7f7a08a86640 10 librados: watch_flush exit 2024-03-29T11:39:47.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.514+0000 7f7a08a86640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T11:39:47.516 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.514+0000 7f7a08a86640 10 client.4406.objecter successfully canceled tick 2024-03-29T11:39:47.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.515+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 >> v1:172.21.15.145:6816/227372239 conn(0x7f79e406dcf0 legacy=0x7f79e40701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:47.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.515+0000 7f7a08a86640 10 monclient: shutdown 2024-03-29T11:39:47.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.515+0000 7f7a08a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:47.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.515+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 >> v1:172.21.15.145:6790/0 conn(0x7f7a040a9ac0 legacy=0x7f7a041d6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:47.518 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.517+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 shutdown_connections 2024-03-29T11:39:47.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.517+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 >> 172.21.15.145:0/2334179402 conn(0x7f7a040b35e0 msgr2=0x7f7a040adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:47.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.517+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 shutdown_connections 2024-03-29T11:39:47.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.517+0000 7f7a08a86640 1 -- 172.21.15.145:0/2334179402 wait complete. 2024-03-29T11:39:47.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:47.517+0000 7f7a08a86640 1 librados: shutdown 2024-03-29T11:39:47.538 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:39:54.344 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:39:54.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.477+0000 7fc15d3c7640 10 monclient: get_monmap_and_config 2024-03-29T11:39:54.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.477+0000 7fc15d3c7640 10 monclient: build_initial_monmap 2024-03-29T11:39:54.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.477+0000 7fc15d3c7640 10 monclient: monmap: 2024-03-29T11:39:54.478 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:54.478 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:54.478 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:39:54.477883+0000 2024-03-29T11:39:54.478 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:39:54.477883+0000 2024-03-29T11:39:54.478 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:54.478 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:54.479 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:54.479 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:39:54.479 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:39:54.479 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:54.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.478+0000 7fc15d3c7640 1 Processor -- start 2024-03-29T11:39:54.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.478+0000 7fc15d3c7640 1 -- start start 2024-03-29T11:39:54.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.478+0000 7fc15d3c7640 10 monclient: init 2024-03-29T11:39:54.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.479+0000 7fc15d3c7640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:54.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.479+0000 7fc15d3c7640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:39:54.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.479+0000 7fc15d3c7640 10 monclient(hunting): picked mon.noname-b con 0x7fc1581422b0 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:54.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.479+0000 7fc15d3c7640 10 monclient(hunting): picked mon.noname-c con 0x7fc158142690 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:54.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.480+0000 7fc1567fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc158142690 0x7fc1580b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:35448/0 (socket says 172.21.15.145:35448) 2024-03-29T11:39:54.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.480+0000 7fc156ffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fc1581422b0 0x7fc15813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:40292/0 (socket says 172.21.15.145:40292) 2024-03-29T11:39:54.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.480+0000 7fc1567fc640 1 -- 172.21.15.145:0/1692778580 learned_addr learned my addr 172.21.15.145:0/1692778580 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:54.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.480+0000 7fc15d3c7640 10 monclient(hunting): picked mon.noname-a con 0x7fc1580b17d0 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:54.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.480+0000 7fc15d3c7640 1 -- 172.21.15.145:0/1692778580 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc158142c80 con 0x7fc1580b17d0 2024-03-29T11:39:54.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.480+0000 7fc15d3c7640 1 -- 172.21.15.145:0/1692778580 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1580aae40 con 0x7fc158142690 2024-03-29T11:39:54.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.480+0000 7fc15d3c7640 1 -- 172.21.15.145:0/1692778580 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15813fbd0 con 0x7fc1581422b0 2024-03-29T11:39:54.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.480+0000 7fc15d3c7640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:54.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.480+0000 7fc15d3c7640 10 monclient(hunting): authenticate will time out at 1183.787867s 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.481+0000 7fc155ffb640 1 -- 172.21.15.145:0/1692778580 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215550038 0 0) 0x7fc15813fbd0 con 0x7fc1581422b0 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.481+0000 7fc155ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.481+0000 7fc155ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.481+0000 7fc155ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.481+0000 7fc155ffb640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.481+0000 7fc155ffb640 1 -- 172.21.15.145:0/1692778580 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc140003520 con 0x7fc1581422b0 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.481+0000 7fc155ffb640 1 -- 172.21.15.145:0/1692778580 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 764492025 0 0) 0x7fc140003520 con 0x7fc1581422b0 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.481+0000 7fc155ffb640 1 -- 172.21.15.145:0/1692778580 >> v1:172.21.15.203:6789/0 conn(0x7fc158142690 legacy=0x7fc1580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.481+0000 7fc155ffb640 1 -- 172.21.15.145:0/1692778580 >> v1:172.21.15.145:6789/0 conn(0x7fc1580b17d0 legacy=0x7fc1580b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 1 monclient: found mon.noname-b 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 1 -- 172.21.15.145:0/1692778580 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc158150930 con 0x7fc1581422b0 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 10 monclient: _finish_auth 0 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:39:54.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 1 -- 172.21.15.145:0/1692778580 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc148001670 con 0x7fc1581422b0 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc155ffb640 10 monclient: dump: 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc15d3c7640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:39:54.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.482+0000 7fc15d3c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:39:54.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.483+0000 7fc155ffb640 1 -- 172.21.15.145:0/1692778580 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc148001890 con 0x7fc1581422b0 2024-03-29T11:39:54.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.483+0000 7fc155ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:54.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.483+0000 7fc15d3c7640 10 monclient: get_monmap_and_config success 2024-03-29T11:39:54.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.483+0000 7fc15d3c7640 4 set_mon_vals no callback set 2024-03-29T11:39:54.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.483+0000 7fc155ffb640 1 -- 172.21.15.145:0/1692778580 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc148001c60 con 0x7fc1581422b0 2024-03-29T11:39:54.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.483+0000 7fc155ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:54.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.483+0000 7fc155ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.483+0000 7fc155ffb640 10 monclient: dump: 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.484+0000 7fc15d3c7640 10 monclient: shutdown 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.484+0000 7fc15d3c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.484+0000 7fc15d3c7640 1 -- 172.21.15.145:0/1692778580 >> v1:172.21.15.145:6790/0 conn(0x7fc1581422b0 legacy=0x7fc15813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.484+0000 7fc15d3c7640 1 -- 172.21.15.145:0/1692778580 shutdown_connections 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.484+0000 7fc15d3c7640 1 -- 172.21.15.145:0/1692778580 >> 172.21.15.145:0/1692778580 conn(0x7fc158066a00 msgr2=0x7fc158066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:54.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.484+0000 7fc15d3c7640 1 -- 172.21.15.145:0/1692778580 shutdown_connections 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.484+0000 7fc15d3c7640 1 -- 172.21.15.145:0/1692778580 wait complete. 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.485+0000 7fc15d3c7640 10 monclient: build_initial_monmap 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.485+0000 7fc15d3c7640 10 monclient: monmap: 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.485+0000 7fc15d3c7640 1 librados: starting msgr at 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.485+0000 7fc15d3c7640 1 librados: starting objecter 2024-03-29T11:39:54.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.486+0000 7fc15d3c7640 1 Processor -- start 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.486+0000 7fc15d3c7640 1 -- start start 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.486+0000 7fc15d3c7640 1 librados: setting wanted keys 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.486+0000 7fc15d3c7640 1 librados: calling monclient init 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.486+0000 7fc15d3c7640 10 monclient: init 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.486+0000 7fc15d3c7640 10 monclient: _reopen_session rank -1 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.486+0000 7fc15d3c7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.486+0000 7fc15d3c7640 10 monclient(hunting): picked mon.noname-b con 0x7fc1580b17d0 addr v1:172.21.15.203:6789/0 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.487+0000 7fc15d3c7640 10 monclient(hunting): picked mon.noname-c con 0x7fc1581422b0 addr v1:172.21.15.145:6790/0 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.487+0000 7fc15d3c7640 10 monclient(hunting): picked mon.noname-a con 0x7fc158142690 addr v1:172.21.15.145:6789/0 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.487+0000 7fc15d3c7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15814e3b0 con 0x7fc158142690 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.487+0000 7fc15d3c7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15812f530 con 0x7fc1580b17d0 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.487+0000 7fc15d3c7640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc15812f810 con 0x7fc1581422b0 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.487+0000 7fc15d3c7640 10 monclient(hunting): _renew_subs 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.487+0000 7fc15d3c7640 10 monclient(hunting): authenticate will time out at 1183.794079s 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.487+0000 7fc156ffd640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fc1580b17d0 0x7fc1580b5320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:36982/0 (socket says 172.21.15.145:36982) 2024-03-29T11:39:54.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.487+0000 7fc156ffd640 1 -- 172.21.15.145:0/2316994985 learned_addr learned my addr 172.21.15.145:0/2316994985 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:39:54.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.488+0000 7fc13f7fe640 10 client.?.objecter ms_handle_connect 0x7fc1581422b0 2024-03-29T11:39:54.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.488+0000 7fc13f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:39:54.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.488+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809528046 0 0) 0x7fc15812f810 con 0x7fc1581422b0 2024-03-29T11:39:54.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.488+0000 7fc13f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:39:54.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.488+0000 7fc13f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:39:54.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.488+0000 7fc13f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.488+0000 7fc13f7fe640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.488+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc144004eb0 con 0x7fc1581422b0 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 5448875 0 0) 0x7fc144004eb0 con 0x7fc1581422b0 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc15812f810 con 0x7fc1581422b0 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc14c0017c0 con 0x7fc1581422b0 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 10 monclient(hunting): dump: 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:54.493 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1541566578 0 0) 0x7fc15812f810 con 0x7fc1581422b0 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 >> v1:172.21.15.203:6789/0 conn(0x7fc1580b17d0 legacy=0x7fc1580b5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.489+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 >> v1:172.21.15.145:6789/0 conn(0x7fc158142690 legacy=0x7fc15814dca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.490+0000 7fc13f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.490+0000 7fc13f7fe640 1 monclient: found mon.c 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.490+0000 7fc13f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.490+0000 7fc13f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.490+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc15812de70 con 0x7fc1581422b0 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc13f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc13f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc13f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc15d3c7640 5 monclient: authenticate success, global_id 4412 2024-03-29T11:39:54.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc15d3c7640 10 monclient: _renew_subs 2024-03-29T11:39:54.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc15d3c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:54.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1580b7240 con 0x7fc1581422b0 2024-03-29T11:39:54.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc15d3c7640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:39:54.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc15d3c7640 10 monclient: _renew_subs 2024-03-29T11:39:54.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc15d3c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:54.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15812d950 con 0x7fc1581422b0 2024-03-29T11:39:54.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.491+0000 7fc15d3c7640 1 librados: init done 2024-03-29T11:39:54.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.492+0000 7fc15d3c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:54.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.492+0000 7fc15d3c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:54.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.492+0000 7fc15d3c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:54.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.492+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc15812e060 con 0x7fc1581422b0 2024-03-29T11:39:54.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.493+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc14c001cf0 con 0x7fc1581422b0 2024-03-29T11:39:54.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.493+0000 7fc13f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:39:54.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.493+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fc14c0049c0 con 0x7fc1581422b0 2024-03-29T11:39:54.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.493+0000 7fc13f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:39:54.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.493+0000 7fc13f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.493+0000 7fc13f7fe640 10 monclient: dump: 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.493+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fc14c004be0 con 0x7fc1581422b0 2024-03-29T11:39:54.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.494+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fc14c084ae0 con 0x7fc1581422b0 2024-03-29T11:39:54.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.494+0000 7fc13f7fe640 10 client.4412.objecter ms_dispatch 0x7fc158000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:39:54.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.494+0000 7fc13f7fe640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:39:54.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.494+0000 7fc13f7fe640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:39:54.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.494+0000 7fc13f7fe640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T11:39:54.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.493+0000 7fc1557fa640 4 set_mon_vals no callback set 2024-03-29T11:39:54.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.494+0000 7fc156ffd640 10 monclient: get_auth_request con 0x7fc14406dcd0 auth_method 0 2024-03-29T11:39:54.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.495+0000 7fc13f7fe640 10 client.4412.objecter ms_handle_connect 0x7fc14406dcd0 2024-03-29T11:39:54.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.501+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc14c050440 con 0x7fc1581422b0 2024-03-29T11:39:54.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.501+0000 7fc13f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:39:54.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.501+0000 7fc13f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:39:54.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.803+0000 7fc15d3c7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:54.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.803+0000 7fc15d3c7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:54.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.803+0000 7fc15d3c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:39:54.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.803+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc158075f10 con 0x7fc1581422b0 2024-03-29T11:39:54.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.804+0000 7fc13f7fe640 1 -- 172.21.15.145:0/2316994985 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fc14c052370 con 0x7fc1581422b0 2024-03-29T11:39:54.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.804+0000 7fc13f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:39:54.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.804+0000 7fc13f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:39:54.805 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:39:54.805 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.807+0000 7fc15d3c7640 10 librados: watch_flush enter 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.807+0000 7fc15d3c7640 10 librados: watch_flush exit 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.807+0000 7fc15d3c7640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.807+0000 7fc15d3c7640 10 client.4412.objecter successfully canceled tick 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.808+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 >> v1:172.21.15.145:6816/227372239 conn(0x7fc14406dcd0 legacy=0x7fc144070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.808+0000 7fc15d3c7640 10 monclient: shutdown 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.808+0000 7fc15d3c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.808+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 >> v1:172.21.15.145:6790/0 conn(0x7fc1581422b0 legacy=0x7fc15814d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.808+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 shutdown_connections 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.808+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 >> 172.21.15.145:0/2316994985 conn(0x7fc158066a00 msgr2=0x7fc1580b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.808+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 shutdown_connections 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.808+0000 7fc15d3c7640 1 -- 172.21.15.145:0/2316994985 wait complete. 2024-03-29T11:39:54.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:39:54.808+0000 7fc15d3c7640 1 librados: shutdown 2024-03-29T11:39:54.822 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:01.629 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:01.775 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.774+0000 7fd8cea51640 10 monclient: get_monmap_and_config 2024-03-29T11:40:01.775 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.774+0000 7fd8cea51640 10 monclient: build_initial_monmap 2024-03-29T11:40:01.775 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.774+0000 7fd8cea51640 10 monclient: monmap: 2024-03-29T11:40:01.775 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:01.775 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:01.775 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:40:01.774358+0000 2024-03-29T11:40:01.776 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:40:01.774358+0000 2024-03-29T11:40:01.776 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:01.776 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:01.776 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:01.776 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:40:01.776 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:40:01.776 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:01.777 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.775+0000 7fd8cea51640 1 Processor -- start 2024-03-29T11:40:01.777 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.775+0000 7fd8cea51640 1 -- start start 2024-03-29T11:40:01.777 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.775+0000 7fd8cea51640 10 monclient: init 2024-03-29T11:40:01.777 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.776+0000 7fd8cea51640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:01.777 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.776+0000 7fd8cea51640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:01.778 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cea51640 10 monclient(hunting): picked mon.noname-a con 0x7fd8c80a9fe0 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:01.778 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cea51640 10 monclient(hunting): picked mon.noname-c con 0x7fd8c80af600 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:01.778 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cea51640 10 monclient(hunting): picked mon.noname-b con 0x7fd8c80ad340 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:01.778 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cea51640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c80aaec0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.778 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cea51640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c80ab6a0 con 0x7fd8c80af600 2024-03-29T11:40:01.778 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cea51640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c80b00a0 con 0x7fd8c80ad340 2024-03-29T11:40:01.778 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cea51640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:01.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cea51640 10 monclient(hunting): authenticate will time out at 1191.084261s 2024-03-29T11:40:01.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cd24e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd8c80af600 0x7fd8c80acc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:42938/0 (socket says 172.21.15.145:42938) 2024-03-29T11:40:01.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cd24e640 1 -- 172.21.15.145:0/3829937430 learned_addr learned my addr 172.21.15.145:0/3829937430 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:01.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cca4d640 1 -- 172.21.15.145:0/3829937430 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034455463 0 0) 0x7fd8c80ab6a0 con 0x7fd8c80af600 2024-03-29T11:40:01.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cca4d640 10 monclient(hunting): _negotiate 2024-03-29T11:40:01.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cca4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:01.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cca4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:01.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cca4d640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:40:01.779 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.777+0000 7fd8cca4d640 1 -- 172.21.15.145:0/3829937430 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b4003520 con 0x7fd8c80af600 2024-03-29T11:40:01.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.778+0000 7fd8cca4d640 1 -- 172.21.15.145:0/3829937430 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1174162113 0 0) 0x7fd8b4003520 con 0x7fd8c80af600 2024-03-29T11:40:01.780 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.778+0000 7fd8cca4d640 1 -- 172.21.15.145:0/3829937430 >> v1:172.21.15.145:6790/0 conn(0x7fd8c80ad340 legacy=0x7fd8c8149c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:01.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 1 -- 172.21.15.145:0/3829937430 >> v1:172.21.15.145:6789/0 conn(0x7fd8c80a9fe0 legacy=0x7fd8c813b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:01.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 10 monclient: _finish_hunting 0 2024-03-29T11:40:01.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 1 monclient: found mon.noname-c 2024-03-29T11:40:01.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:01.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:40:01.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 1 -- 172.21.15.145:0/3829937430 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c813beb0 con 0x7fd8c80af600 2024-03-29T11:40:01.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 10 monclient: _finish_auth 0 2024-03-29T11:40:01.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 10 monclient: _check_auth_tickets 2024-03-29T11:40:01.783 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:01.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 1 -- 172.21.15.145:0/3829937430 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd8b8001690 con 0x7fd8c80af600 2024-03-29T11:40:01.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cea51640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:40:01.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cea51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:01.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:01.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:01.784 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 10 monclient: dump: 2024-03-29T11:40:01.784 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:01.784 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:01.784 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:01.785 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:01.785 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:01.785 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:01.785 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:01.785 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:01.785 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:01.785 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 1 -- 172.21.15.145:0/3829937430 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8b80019a0 con 0x7fd8c80af600 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cea51640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cca4d640 1 -- 172.21.15.145:0/3829937430 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd8b8001d30 con 0x7fd8c80af600 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cea51640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.781+0000 7fd8cea51640 4 set_mon_vals no callback set 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.782+0000 7fd8cca4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.782+0000 7fd8cca4d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.782+0000 7fd8cca4d640 10 monclient: dump: 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:01.786 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:01.787 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:01.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.782+0000 7fd8cea51640 10 monclient: shutdown 2024-03-29T11:40:01.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.782+0000 7fd8cea51640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:01.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.782+0000 7fd8cea51640 1 -- 172.21.15.145:0/3829937430 >> v1:172.21.15.203:6789/0 conn(0x7fd8c80af600 legacy=0x7fd8c80acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:01.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.783+0000 7fd8cea51640 1 -- 172.21.15.145:0/3829937430 shutdown_connections 2024-03-29T11:40:01.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.783+0000 7fd8cea51640 1 -- 172.21.15.145:0/3829937430 >> 172.21.15.145:0/3829937430 conn(0x7fd8c80673e0 msgr2=0x7fd8c80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:01.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.783+0000 7fd8cea51640 1 -- 172.21.15.145:0/3829937430 shutdown_connections 2024-03-29T11:40:01.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.783+0000 7fd8cea51640 1 -- 172.21.15.145:0/3829937430 wait complete. 2024-03-29T11:40:01.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.784+0000 7fd8cea51640 10 monclient: build_initial_monmap 2024-03-29T11:40:01.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.784+0000 7fd8cea51640 10 monclient: monmap: 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.784+0000 7fd8cea51640 1 librados: starting msgr at 2024-03-29T11:40:01.792 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.784+0000 7fd8cea51640 1 librados: starting objecter 2024-03-29T11:40:01.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.784+0000 7fd8cea51640 1 Processor -- start 2024-03-29T11:40:01.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.784+0000 7fd8cea51640 1 -- start start 2024-03-29T11:40:01.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.784+0000 7fd8cea51640 1 librados: setting wanted keys 2024-03-29T11:40:01.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.784+0000 7fd8cea51640 1 librados: calling monclient init 2024-03-29T11:40:01.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.784+0000 7fd8cea51640 10 monclient: init 2024-03-29T11:40:01.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:01.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:40:01.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 10 monclient(hunting): picked mon.noname-b con 0x7fd8c80a9fe0 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:01.793 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 10 monclient(hunting): picked mon.noname-a con 0x7fd8c80ad340 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 10 monclient(hunting): picked mon.noname-c con 0x7fd8c80af600 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c8071cf0 con 0x7fd8c80ad340 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c80710e0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c80713c0 con 0x7fd8c80af600 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.785+0000 7fd8cea51640 10 monclient(hunting): authenticate will time out at 1191.092803s 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.786+0000 7fd8cda4f640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fd8c80a9fe0 0x7fd8c8072180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:43272/0 (socket says 172.21.15.145:43272) 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.786+0000 7fd8cda4f640 1 -- 172.21.15.145:0/1113013804 learned_addr learned my addr 172.21.15.145:0/1113013804 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.787+0000 7fd8be7fc640 10 client.?.objecter ms_handle_connect 0x7fd8c80a9fe0 2024-03-29T11:40:01.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.787+0000 7fd8be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.787+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098138400 0 0) 0x7fd8c80710e0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.787+0000 7fd8be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.787+0000 7fd8be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.787+0000 7fd8be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.787+0000 7fd8be7fc640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.787+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ac004eb0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 737752232 0 0) 0x7fd8ac004eb0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8c80710e0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 client.?.objecter ms_handle_connect 0x7fd8c80af600 2024-03-29T11:40:01.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd8c0001930 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 monclient(hunting): dump: 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:01.796 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176610713 0 0) 0x7fd8c80713c0 con 0x7fd8c80af600 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ac004eb0 con 0x7fd8c80af600 2024-03-29T11:40:01.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3054555231 0 0) 0x7fd8c80710e0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 >> v1:172.21.15.145:6790/0 conn(0x7fd8c80af600 legacy=0x7fd8c81e5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:01.798 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.788+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 >> v1:172.21.15.145:6789/0 conn(0x7fd8c80ad340 legacy=0x7fd8c806f9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:01.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:01.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8be7fc640 1 monclient: found mon.b 2024-03-29T11:40:01.801 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c80716a0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8cea51640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8cea51640 10 monclient: _renew_subs 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8cea51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8cea51640 1 -- 172.21.15.145:0/1113013804 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c806f040 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8cea51640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8cea51640 10 monclient: _renew_subs 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.792+0000 7fd8cea51640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:40:01.802 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.793+0000 7fd8cea51640 1 -- 172.21.15.145:0/1113013804 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c813e840 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.793+0000 7fd8cea51640 1 librados: init done 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.793+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8c0001d60 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.793+0000 7fd8be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.793+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd8c0003a60 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.793+0000 7fd8be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.793+0000 7fd8be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.793+0000 7fd8be7fc640 10 monclient: dump: 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:01.803 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.793+0000 7fd8bffff640 4 set_mon_vals no callback set 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.795+0000 7fd8abfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.795+0000 7fd8abfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.795+0000 7fd8abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.795+0000 7fd8abfff640 1 -- 172.21.15.145:0/1113013804 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c804f730 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.796+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fd8c0003bc0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.797+0000 7fd8cd24e640 10 monclient: get_auth_request con 0x7fd8ac06dc60 auth_method 0 2024-03-29T11:40:01.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.797+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fd8c0084ad0 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.797+0000 7fd8be7fc640 10 client.4411.objecter ms_dispatch 0x7fd8c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:01.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.797+0000 7fd8be7fc640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:01.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.797+0000 7fd8be7fc640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:01.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.797+0000 7fd8be7fc640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:40:01.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.797+0000 7fd8be7fc640 10 client.4411.objecter ms_handle_connect 0x7fd8ac06dc60 2024-03-29T11:40:01.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.805+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8c0086e60 con 0x7fd8c80a9fe0 2024-03-29T11:40:01.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.805+0000 7fd8be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:01.808 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:01.805+0000 7fd8be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:02.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.125+0000 7fd8abfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:02.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.125+0000 7fd8abfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:02.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.125+0000 7fd8abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:40:02.126 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.125+0000 7fd8abfff640 1 -- 172.21.15.145:0/1113013804 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8c8074e10 con 0x7fd8c80a9fe0 2024-03-29T11:40:02.128 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:40:02.128 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:02.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.127+0000 7fd8be7fc640 1 -- 172.21.15.145:0/1113013804 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fd8c00501e0 con 0x7fd8c80a9fe0 2024-03-29T11:40:02.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.127+0000 7fd8be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:02.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.127+0000 7fd8be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:02.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.131+0000 7fd8abfff640 10 librados: watch_flush enter 2024-03-29T11:40:02.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.132+0000 7fd8abfff640 10 librados: watch_flush exit 2024-03-29T11:40:02.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.132+0000 7fd8abfff640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:40:02.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.132+0000 7fd8abfff640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:40:02.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.134+0000 7fd8abfff640 1 -- 172.21.15.145:0/1113013804 >> v1:172.21.15.145:6816/227372239 conn(0x7fd8ac06dc60 legacy=0x7fd8ac070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:02.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.134+0000 7fd8abfff640 10 monclient: shutdown 2024-03-29T11:40:02.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.134+0000 7fd8abfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:02.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.134+0000 7fd8abfff640 1 -- 172.21.15.145:0/1113013804 >> v1:172.21.15.203:6789/0 conn(0x7fd8c80a9fe0 legacy=0x7fd8c8072180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:02.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.134+0000 7fd8abfff640 1 -- 172.21.15.145:0/1113013804 shutdown_connections 2024-03-29T11:40:02.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.134+0000 7fd8abfff640 1 -- 172.21.15.145:0/1113013804 >> 172.21.15.145:0/1113013804 conn(0x7fd8c80673e0 msgr2=0x7fd8c80aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:02.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.134+0000 7fd8abfff640 1 -- 172.21.15.145:0/1113013804 shutdown_connections 2024-03-29T11:40:02.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.134+0000 7fd8abfff640 1 -- 172.21.15.145:0/1113013804 wait complete. 2024-03-29T11:40:02.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:02.135+0000 7fd8abfff640 1 librados: shutdown 2024-03-29T11:40:02.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:08.961 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.104+0000 7f53729e8640 10 monclient: get_monmap_and_config 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.104+0000 7f53729e8640 10 monclient: build_initial_monmap 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.104+0000 7f53729e8640 10 monclient: monmap: 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:40:09.106007+0000 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:40:09.106007+0000 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:09.108 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:40:09.109 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:40:09.109 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:09.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.108+0000 7f53729e8640 1 Processor -- start 2024-03-29T11:40:09.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.108+0000 7f53729e8640 1 -- start start 2024-03-29T11:40:09.110 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.108+0000 7f53729e8640 10 monclient: init 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 10 monclient(hunting): picked mon.noname-c con 0x7f536c0a9750 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 10 monclient(hunting): picked mon.noname-b con 0x7f536c0af100 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 10 monclient(hunting): picked mon.noname-a con 0x7f536c13d200 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c06dc20 con 0x7f536c13d200 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c06df00 con 0x7f536c0a9750 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c13d710 con 0x7f536c0af100 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:09.115 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.109+0000 7f53729e8640 10 monclient(hunting): authenticate will time out at 1198.417463s 2024-03-29T11:40:09.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.113+0000 7f53711e5640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f536c0af100 0x7f536c143250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:44607/0 (socket says 172.21.15.145:44607) 2024-03-29T11:40:09.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.113+0000 7f53711e5640 1 -- 172.21.15.145:0/3898003408 learned_addr learned my addr 172.21.15.145:0/3898003408 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:09.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508144542 0 0) 0x7f536c06dc20 con 0x7f536c13d200 2024-03-29T11:40:09.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 10 monclient(hunting): _negotiate 2024-03-29T11:40:09.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:09.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:09.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:40:09.116 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5358003520 con 0x7f536c13d200 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105028908 0 0) 0x7f536c13d710 con 0x7f536c0af100 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 10 monclient(hunting): _negotiate 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.114+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536c06dc20 con 0x7f536c0af100 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 380545110 0 0) 0x7f536c06dc20 con 0x7f536c0af100 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 >> v1:172.21.15.203:6789/0 conn(0x7f536c0a9750 legacy=0x7f536c13caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 >> v1:172.21.15.145:6789/0 conn(0x7f536c13d200 legacy=0x7f536c143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:09.117 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 10 monclient: _finish_hunting 0 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 1 monclient: found mon.noname-b 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c0af650 con 0x7f536c0af100 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 10 monclient: _finish_auth 0 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 10 monclient: _check_auth_tickets 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5368001670 con 0x7f536c0af100 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53729e8640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53729e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:09.118 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53709e4640 10 monclient: dump: 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:09.119 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.115+0000 7f53729e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:09.120 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.118+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53680017d0 con 0x7f536c0af100 2024-03-29T11:40:09.120 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.118+0000 7f53709e4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:09.120 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.118+0000 7f53709e4640 1 -- 172.21.15.145:0/3898003408 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5368001be0 con 0x7f536c0af100 2024-03-29T11:40:09.120 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.118+0000 7f53709e4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:09.120 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.118+0000 7f53709e4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:09.120 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.118+0000 7f53709e4640 10 monclient: dump: 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.118+0000 7f53729e8640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:09.121 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.118+0000 7f53729e8640 4 set_mon_vals no callback set 2024-03-29T11:40:09.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.119+0000 7f53729e8640 10 monclient: shutdown 2024-03-29T11:40:09.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.119+0000 7f53729e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:09.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.119+0000 7f53729e8640 1 -- 172.21.15.145:0/3898003408 >> v1:172.21.15.145:6790/0 conn(0x7f536c0af100 legacy=0x7f536c143250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:09.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.120+0000 7f53729e8640 1 -- 172.21.15.145:0/3898003408 shutdown_connections 2024-03-29T11:40:09.122 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.120+0000 7f53729e8640 1 -- 172.21.15.145:0/3898003408 >> 172.21.15.145:0/3898003408 conn(0x7f536c0b3a30 msgr2=0x7f536c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:09.123 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.121+0000 7f53729e8640 1 -- 172.21.15.145:0/3898003408 shutdown_connections 2024-03-29T11:40:09.123 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.122+0000 7f53729e8640 1 -- 172.21.15.145:0/3898003408 wait complete. 2024-03-29T11:40:09.124 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.122+0000 7f53729e8640 10 monclient: build_initial_monmap 2024-03-29T11:40:09.124 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.122+0000 7f53729e8640 10 monclient: monmap: 2024-03-29T11:40:09.124 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:09.124 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:09.124 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:40:09.124 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:40:09.124 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:09.125 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:09.125 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:09.125 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:40:09.125 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:40:09.125 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:09.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.123+0000 7f53729e8640 1 librados: starting msgr at 2024-03-29T11:40:09.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.123+0000 7f53729e8640 1 librados: starting objecter 2024-03-29T11:40:09.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.123+0000 7f53729e8640 1 Processor -- start 2024-03-29T11:40:09.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.123+0000 7f53729e8640 1 -- start start 2024-03-29T11:40:09.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.123+0000 7f53729e8640 1 librados: setting wanted keys 2024-03-29T11:40:09.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.123+0000 7f53729e8640 1 librados: calling monclient init 2024-03-29T11:40:09.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.123+0000 7f53729e8640 10 monclient: init 2024-03-29T11:40:09.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.124+0000 7f53729e8640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:09.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.124+0000 7f53729e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:09.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.127+0000 7f53729e8640 10 monclient(hunting): picked mon.noname-a con 0x7f536c0a9750 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:09.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.127+0000 7f53719e6640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f536c0a9750 0x7f536c134950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36073/0 (socket says 172.21.15.145:36073) 2024-03-29T11:40:09.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.130+0000 7f53729e8640 10 monclient(hunting): picked mon.noname-c con 0x7f536c13d200 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.130+0000 7f53729e8640 10 monclient(hunting): picked mon.noname-b con 0x7f536c142c70 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.130+0000 7f53729e8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c13e080 con 0x7f536c0a9750 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.130+0000 7f53729e8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c13f7d0 con 0x7f536c142c70 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.130+0000 7f53729e8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c135060 con 0x7f536c13d200 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.130+0000 7f53729e8640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:09.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.130+0000 7f53729e8640 10 monclient(hunting): authenticate will time out at 1198.438523s 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.130+0000 7f53719e6640 1 -- 172.21.15.145:0/1567534883 learned_addr learned my addr 172.21.15.145:0/1567534883 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.131+0000 7f53627fc640 10 client.?.objecter ms_handle_connect 0x7f536c13d200 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.131+0000 7f53627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.131+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3421732061 0 0) 0x7f536c135060 con 0x7f536c13d200 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.131+0000 7f53627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.131+0000 7f53627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.131+0000 7f53627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.131+0000 7f53627fc640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:40:09.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.131+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5350004eb0 con 0x7f536c13d200 2024-03-29T11:40:09.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.132+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2821144942 0 0) 0x7f5350004eb0 con 0x7f536c13d200 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.132+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f536c135060 con 0x7f536c13d200 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.132+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5368001b20 con 0x7f536c13d200 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.132+0000 7f53627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.132+0000 7f53627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.132+0000 7f53627fc640 10 monclient(hunting): dump: 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:09.136 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.132+0000 7f53627fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.132+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 656481435 0 0) 0x7f536c135060 con 0x7f536c13d200 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.132+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 >> v1:172.21.15.203:6789/0 conn(0x7f536c142c70 legacy=0x7f536c1d3220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 >> v1:172.21.15.145:6789/0 conn(0x7f536c0a9750 legacy=0x7f536c134950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 1 monclient: found mon.c 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c1351f0 con 0x7f536c13d200 2024-03-29T11:40:09.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 10 client.?.objecter ms_handle_connect 0x7f536c0a9750 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53729e8640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53729e8640 10 monclient: _renew_subs 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53729e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53729e8640 1 -- 172.21.15.145:0/1567534883 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c0ac040 con 0x7f536c13d200 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53729e8640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53729e8640 10 monclient: _renew_subs 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53729e8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53729e8640 1 -- 172.21.15.145:0/1567534883 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c13d9d0 con 0x7f536c13d200 2024-03-29T11:40:09.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.133+0000 7f53729e8640 1 librados: init done 2024-03-29T11:40:09.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.135+0000 7f5357fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.135+0000 7f5357fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.135+0000 7f5357fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.135+0000 7f5357fff640 1 -- 172.21.15.145:0/1567534883 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536c1353e0 con 0x7f536c13d200 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.136+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5368004a90 con 0x7f536c13d200 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.136+0000 7f53627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.136+0000 7f5363fff640 4 set_mon_vals no callback set 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.136+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5368004e60 con 0x7f536c13d200 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.136+0000 7f53627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.136+0000 7f53627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.136+0000 7f53627fc640 10 monclient: dump: 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:09.141 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.136+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f5368005080 con 0x7f536c13d200 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.137+0000 7f53719e6640 10 monclient: get_auth_request con 0x7f535006dd70 auth_method 0 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.137+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f5368084b20 con 0x7f536c13d200 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.137+0000 7f53627fc640 10 client.4421.objecter ms_dispatch 0x7f536c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.137+0000 7f53627fc640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.137+0000 7f53627fc640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:09.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.137+0000 7f53627fc640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-29T11:40:09.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.137+0000 7f53627fc640 10 client.4421.objecter ms_handle_connect 0x7f535006dd70 2024-03-29T11:40:09.146 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.144+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5368086e60 con 0x7f536c13d200 2024-03-29T11:40:09.146 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.144+0000 7f53627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:09.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.144+0000 7f53627fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:09.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.476+0000 7f5357fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:09.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.476+0000 7f5357fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:09.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.476+0000 7f5357fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:09.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.476+0000 7f5357fff640 1 -- 172.21.15.145:0/1567534883 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f536c13e7e0 con 0x7f536c13d200 2024-03-29T11:40:09.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.479+0000 7f53627fc640 1 -- 172.21.15.145:0/1567534883 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f5368050540 con 0x7f536c13d200 2024-03-29T11:40:09.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.479+0000 7f53627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:09.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.479+0000 7f53627fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:09.482 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:40:09.482 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:09.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.483+0000 7f5357fff640 10 librados: watch_flush enter 2024-03-29T11:40:09.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.483+0000 7f5357fff640 10 librados: watch_flush exit 2024-03-29T11:40:09.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.483+0000 7f5357fff640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-29T11:40:09.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.483+0000 7f5357fff640 10 client.4421.objecter successfully canceled tick 2024-03-29T11:40:09.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.484+0000 7f5357fff640 1 -- 172.21.15.145:0/1567534883 >> v1:172.21.15.145:6816/227372239 conn(0x7f535006dd70 legacy=0x7f5350070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:09.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.484+0000 7f5357fff640 10 monclient: shutdown 2024-03-29T11:40:09.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.484+0000 7f5357fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:09.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.484+0000 7f5357fff640 1 -- 172.21.15.145:0/1567534883 >> v1:172.21.15.145:6790/0 conn(0x7f536c13d200 legacy=0x7f536c1d2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:09.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.485+0000 7f5357fff640 1 -- 172.21.15.145:0/1567534883 shutdown_connections 2024-03-29T11:40:09.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.485+0000 7f5357fff640 1 -- 172.21.15.145:0/1567534883 >> 172.21.15.145:0/1567534883 conn(0x7f536c0b3a30 msgr2=0x7f536c0b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:09.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.486+0000 7f5357fff640 1 -- 172.21.15.145:0/1567534883 shutdown_connections 2024-03-29T11:40:09.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.486+0000 7f5357fff640 1 -- 172.21.15.145:0/1567534883 wait complete. 2024-03-29T11:40:09.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:09.487+0000 7f5357fff640 1 librados: shutdown 2024-03-29T11:40:09.506 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:16.313 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:16.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.465+0000 7f5ac30d7640 10 monclient: get_monmap_and_config 2024-03-29T11:40:16.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.465+0000 7f5ac30d7640 10 monclient: build_initial_monmap 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.465+0000 7f5ac30d7640 10 monclient: monmap: 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:40:16.466385+0000 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:40:16.466385+0000 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:40:16.468 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.466+0000 7f5ac30d7640 1 Processor -- start 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.466+0000 7f5ac30d7640 1 -- start start 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.466+0000 7f5ac30d7640 10 monclient: init 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 10 monclient(hunting): picked mon.noname-b con 0x7f5abc0aac30 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 10 monclient(hunting): picked mon.noname-a con 0x7f5abc072f20 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 10 monclient(hunting): picked mon.noname-c con 0x7f5abc0a9e90 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5abc0ad8e0 con 0x7f5abc072f20 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5abc0ac6a0 con 0x7f5abc0a9e90 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5abc0ab230 con 0x7f5abc0aac30 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:16.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.467+0000 7f5ac30d7640 10 monclient(hunting): authenticate will time out at 1205.775488s 2024-03-29T11:40:16.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.471+0000 7f5ac20d5640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f5abc0aac30 0x7f5abc13af50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:35649/0 (socket says 172.21.15.145:35649) 2024-03-29T11:40:16.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.471+0000 7f5ac20d5640 1 -- 172.21.15.145:0/2042893919 learned_addr learned my addr 172.21.15.145:0/2042893919 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:16.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2928637254 0 0) 0x7f5abc0ab230 con 0x7f5abc0aac30 2024-03-29T11:40:16.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 10 monclient(hunting): _negotiate 2024-03-29T11:40:16.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:16.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:16.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa4003520 con 0x7f5abc0aac30 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462418018 0 0) 0x7f5abc0ac6a0 con 0x7f5abc0a9e90 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 10 monclient(hunting): _negotiate 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.475+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5abc0ab230 con 0x7f5abc0a9e90 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 66544884 0 0) 0x7f5abc0ab230 con 0x7f5abc0a9e90 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 >> v1:172.21.15.145:6790/0 conn(0x7f5abc0aac30 legacy=0x7f5abc13af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 >> v1:172.21.15.145:6789/0 conn(0x7f5abc072f20 legacy=0x7f5abc0acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 10 monclient: _finish_hunting 0 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 1 monclient: found mon.noname-c 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:16.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc13b660 con 0x7f5abc0a9e90 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 10 monclient: _finish_auth 0 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 10 monclient: _check_auth_tickets 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5aac0019f0 con 0x7f5abc0a9e90 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 10 monclient: dump: 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:16.482 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.476+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5aac001b50 con 0x7f5abc0a9e90 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.477+0000 7f5ac10d3640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.477+0000 7f5ac10d3640 1 -- 172.21.15.145:0/2042893919 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5aac001fa0 con 0x7f5abc0a9e90 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.477+0000 7f5ac10d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.477+0000 7f5ac10d3640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.477+0000 7f5ac10d3640 10 monclient: dump: 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:16.483 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:16.484 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.484 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.484 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:16.484 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:16.484 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:16.484 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:16.484 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:16.484 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.479+0000 7f5ac30d7640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.479+0000 7f5ac30d7640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.479+0000 7f5ac30d7640 4 set_mon_vals no callback set 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.480+0000 7f5ac30d7640 10 monclient: shutdown 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.480+0000 7f5ac30d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.480+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2042893919 >> v1:172.21.15.203:6789/0 conn(0x7f5abc0a9e90 legacy=0x7f5abc0ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.480+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2042893919 shutdown_connections 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.480+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2042893919 >> 172.21.15.145:0/2042893919 conn(0x7f5abc0b3a30 msgr2=0x7f5abc0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.481+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2042893919 shutdown_connections 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.481+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2042893919 wait complete. 2024-03-29T11:40:16.486 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.481+0000 7f5ac30d7640 10 monclient: build_initial_monmap 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.481+0000 7f5ac30d7640 10 monclient: monmap: 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.482+0000 7f5ac30d7640 1 librados: starting msgr at 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.482+0000 7f5ac30d7640 1 librados: starting objecter 2024-03-29T11:40:16.487 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.482+0000 7f5ac30d7640 1 Processor -- start 2024-03-29T11:40:16.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.482+0000 7f5ac30d7640 1 -- start start 2024-03-29T11:40:16.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.482+0000 7f5ac30d7640 1 librados: setting wanted keys 2024-03-29T11:40:16.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.482+0000 7f5ac30d7640 1 librados: calling monclient init 2024-03-29T11:40:16.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.482+0000 7f5ac30d7640 10 monclient: init 2024-03-29T11:40:16.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.483+0000 7f5ac30d7640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:16.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.483+0000 7f5ac30d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:16.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.483+0000 7f5ac30d7640 10 monclient(hunting): picked mon.noname-a con 0x7f5abc072f20 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:16.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.486+0000 7f5ac30d7640 10 monclient(hunting): picked mon.noname-b con 0x7f5abc0a9e90 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:16.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5ac30d7640 10 monclient(hunting): picked mon.noname-c con 0x7f5abc0aac30 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:16.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5ac30d7640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5abc1d5dd0 con 0x7f5abc072f20 2024-03-29T11:40:16.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5ac30d7640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5abc12fc10 con 0x7f5abc0a9e90 2024-03-29T11:40:16.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5ac30d7640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5abc1d6c50 con 0x7f5abc0aac30 2024-03-29T11:40:16.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5ac30d7640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:16.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5ac30d7640 10 monclient(hunting): authenticate will time out at 1205.795616s 2024-03-29T11:40:16.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5ac20d5640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5abc072f20 0x7f5abc12d740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:40873/0 (socket says 172.21.15.145:40873) 2024-03-29T11:40:16.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5ac20d5640 1 -- 172.21.15.145:0/2786992697 learned_addr learned my addr 172.21.15.145:0/2786992697 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:16.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5abaffd640 10 client.?.objecter ms_handle_connect 0x7f5abc072f20 2024-03-29T11:40:16.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.487+0000 7f5abaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.488+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079434581 0 0) 0x7f5abc1d5dd0 con 0x7f5abc072f20 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.488+0000 7f5abaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.488+0000 7f5abaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.488+0000 7f5abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.488+0000 7f5abaffd640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.488+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa8004eb0 con 0x7f5abc072f20 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.490+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 705128358 0 0) 0x7f5aa8004eb0 con 0x7f5abc072f20 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.490+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aa8005300 con 0x7f5abc072f20 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.490+0000 7f5abaffd640 10 client.?.objecter ms_handle_connect 0x7f5abc0aac30 2024-03-29T11:40:16.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.490+0000 7f5abaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ab0001850 con 0x7f5abc072f20 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): dump: 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:16.494 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:16.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:16.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2607568568 0 0) 0x7f5abc1d6c50 con 0x7f5abc0aac30 2024-03-29T11:40:16.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:16.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:16.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:16.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:40:16.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa80058a0 con 0x7f5abc0aac30 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2362334176 0 0) 0x7f5aa80058a0 con 0x7f5abc0aac30 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5abc1d6c50 con 0x7f5abc0aac30 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5aac001e60 con 0x7f5abc0aac30 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient(hunting): dump: 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:16.498 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3988836812 0 0) 0x7f5aa8005300 con 0x7f5abc072f20 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 >> v1:172.21.15.145:6790/0 conn(0x7f5abc0aac30 legacy=0x7f5abc130900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 >> v1:172.21.15.203:6789/0 conn(0x7f5abc0a9e90 legacy=0x7f5abc1d6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 monclient: found mon.a 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc1d6020 con 0x7f5abc072f20 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:16.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5abaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5ac30d7640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5ac30d7640 10 monclient: _renew_subs 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5ac30d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5abc131010 con 0x7f5abc072f20 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.491+0000 7f5ac30d7640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5ac30d7640 10 monclient: _renew_subs 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ab00047e0 con 0x7f5abc072f20 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5ac30d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5abc12e560 con 0x7f5abc072f20 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5abaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ab0001850 con 0x7f5abc072f20 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5abaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5abaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:16.500 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5abaffd640 10 monclient: dump: 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5ac08d2640 4 set_mon_vals no callback set 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.492+0000 7f5ac30d7640 1 librados: init done 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.493+0000 7f5ac30d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.493+0000 7f5ac30d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:16.501 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.493+0000 7f5ac30d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:16.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.494+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5abc1d6260 con 0x7f5abc072f20 2024-03-29T11:40:16.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.494+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f5ab001a7c0 con 0x7f5abc072f20 2024-03-29T11:40:16.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.495+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f5ab0084750 con 0x7f5abc072f20 2024-03-29T11:40:16.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.495+0000 7f5abaffd640 10 client.4416.objecter ms_dispatch 0x7f5abc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:16.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.495+0000 7f5abaffd640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:16.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.495+0000 7f5abaffd640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:16.502 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.495+0000 7f5abaffd640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:40:16.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.495+0000 7f5ac18d4640 10 monclient: get_auth_request con 0x7f5aa806e240 auth_method 0 2024-03-29T11:40:16.503 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.495+0000 7f5abaffd640 10 client.4416.objecter ms_handle_connect 0x7f5aa806e240 2024-03-29T11:40:16.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.503+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ab0050090 con 0x7f5abc072f20 2024-03-29T11:40:16.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.503+0000 7f5abaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:16.505 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.503+0000 7f5abaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:16.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.823+0000 7f5ac30d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:16.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.823+0000 7f5ac30d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:16.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.823+0000 7f5ac30d7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:16.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.823+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5abc04f730 con 0x7f5abc072f20 2024-03-29T11:40:16.827 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:40:16.827 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:16.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.824+0000 7f5abaffd640 1 -- 172.21.15.145:0/2786992697 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f5ab0051fc0 con 0x7f5abc072f20 2024-03-29T11:40:16.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.824+0000 7f5abaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:16.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.824+0000 7f5abaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:16.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.827+0000 7f5ac30d7640 10 librados: watch_flush enter 2024-03-29T11:40:16.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.827+0000 7f5ac30d7640 10 librados: watch_flush exit 2024-03-29T11:40:16.829 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.827+0000 7f5ac30d7640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:40:16.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.827+0000 7f5ac30d7640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:40:16.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.828+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 >> v1:172.21.15.145:6816/227372239 conn(0x7f5aa806e240 legacy=0x7f5aa8070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.828+0000 7f5ac30d7640 10 monclient: shutdown 2024-03-29T11:40:16.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.828+0000 7f5ac30d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:16.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.828+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 >> v1:172.21.15.145:6789/0 conn(0x7f5abc072f20 legacy=0x7f5abc12d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:16.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.828+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 shutdown_connections 2024-03-29T11:40:16.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.828+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 >> 172.21.15.145:0/2786992697 conn(0x7f5abc0b3a30 msgr2=0x7f5abc0aeaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:16.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.831+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 shutdown_connections 2024-03-29T11:40:16.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.831+0000 7f5ac30d7640 1 -- 172.21.15.145:0/2786992697 wait complete. 2024-03-29T11:40:16.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:16.831+0000 7f5ac30d7640 1 librados: shutdown 2024-03-29T11:40:16.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:23.658 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.807+0000 7efc5a089640 10 monclient: get_monmap_and_config 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.807+0000 7efc5a089640 10 monclient: build_initial_monmap 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.807+0000 7efc5a089640 10 monclient: monmap: 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:40:23.808538+0000 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:40:23.808538+0000 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:23.814 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.808+0000 7efc5a089640 1 Processor -- start 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.808+0000 7efc5a089640 1 -- start start 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.808+0000 7efc5a089640 10 monclient: init 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 10 monclient(hunting): picked mon.noname-a con 0x7efc540ad6c0 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 10 monclient(hunting): picked mon.noname-b con 0x7efc540af6c0 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 10 monclient(hunting): picked mon.noname-c con 0x7efc5414fdc0 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc540a94e0 con 0x7efc540ad6c0 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc540af3a0 con 0x7efc5414fdc0 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc540afb60 con 0x7efc540af6c0 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:23.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.809+0000 7efc5a089640 10 monclient(hunting): authenticate will time out at 1213.117748s 2024-03-29T11:40:23.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.814+0000 7efc537fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7efc540ad6c0 0x7efc5414f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:58467/0 (socket says 172.21.15.145:58467) 2024-03-29T11:40:23.817 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.814+0000 7efc537fe640 1 -- 172.21.15.145:0/572552864 learned_addr learned my addr 172.21.15.145:0/572552864 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:23.821 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1897092471 0 0) 0x7efc540a94e0 con 0x7efc540ad6c0 2024-03-29T11:40:23.821 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc3c003520 con 0x7efc540ad6c0 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2856382109 0 0) 0x7efc540afb60 con 0x7efc540af6c0 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient(hunting): _negotiate 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc540a94e0 con 0x7efc540af6c0 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2548261013 0 0) 0x7efc540a94e0 con 0x7efc540af6c0 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 >> v1:172.21.15.203:6789/0 conn(0x7efc5414fdc0 legacy=0x7efc5415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 >> v1:172.21.15.145:6789/0 conn(0x7efc540ad6c0 legacy=0x7efc5414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.822 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient: _finish_hunting 0 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 monclient: found mon.noname-b 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc540aab80 con 0x7efc540af6c0 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient: _finish_auth 0 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient: _check_auth_tickets 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efc44001670 con 0x7efc540af6c0 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.816+0000 7efc527fc640 10 monclient: dump: 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:23.823 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.817+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc440017d0 con 0x7efc540af6c0 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.817+0000 7efc527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.817+0000 7efc527fc640 1 -- 172.21.15.145:0/572552864 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efc44001ba0 con 0x7efc540af6c0 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.817+0000 7efc527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.817+0000 7efc527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.817+0000 7efc527fc640 10 monclient: dump: 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:23.824 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.817+0000 7efc5a089640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.817+0000 7efc5a089640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.817+0000 7efc5a089640 4 set_mon_vals no callback set 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.818+0000 7efc5a089640 10 monclient: shutdown 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.818+0000 7efc5a089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.818+0000 7efc5a089640 1 -- 172.21.15.145:0/572552864 >> v1:172.21.15.145:6790/0 conn(0x7efc540af6c0 legacy=0x7efc54152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.819+0000 7efc5a089640 1 -- 172.21.15.145:0/572552864 shutdown_connections 2024-03-29T11:40:23.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.819+0000 7efc5a089640 1 -- 172.21.15.145:0/572552864 >> 172.21.15.145:0/572552864 conn(0x7efc540b35e0 msgr2=0x7efc540b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:23.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.820+0000 7efc5a089640 1 -- 172.21.15.145:0/572552864 shutdown_connections 2024-03-29T11:40:23.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.820+0000 7efc5a089640 1 -- 172.21.15.145:0/572552864 wait complete. 2024-03-29T11:40:23.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.821+0000 7efc5a089640 10 monclient: build_initial_monmap 2024-03-29T11:40:23.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.821+0000 7efc5a089640 10 monclient: monmap: 2024-03-29T11:40:23.826 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:23.826 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:23.826 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:40:23.826 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.822+0000 7efc5a089640 1 librados: starting msgr at 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.822+0000 7efc5a089640 1 librados: starting objecter 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.822+0000 7efc5a089640 1 Processor -- start 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.822+0000 7efc5a089640 1 -- start start 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.822+0000 7efc5a089640 1 librados: setting wanted keys 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.822+0000 7efc5a089640 1 librados: calling monclient init 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.822+0000 7efc5a089640 10 monclient: init 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 10 monclient(hunting): picked mon.noname-c con 0x7efc540ad6c0 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:23.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 10 monclient(hunting): picked mon.noname-b con 0x7efc540af6c0 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:23.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 10 monclient(hunting): picked mon.noname-a con 0x7efc5414fdc0 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:23.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc541e3e60 con 0x7efc5414fdc0 2024-03-29T11:40:23.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc540706e0 con 0x7efc540af6c0 2024-03-29T11:40:23.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc541e7f50 con 0x7efc540ad6c0 2024-03-29T11:40:23.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:23.828 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.823+0000 7efc5a089640 10 monclient(hunting): authenticate will time out at 1213.131270s 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.828+0000 7efc537fe640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7efc540ad6c0 0x7efc541e7840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:36761/0 (socket says 172.21.15.145:36761) 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.828+0000 7efc537fe640 1 -- 172.21.15.145:0/2065598484 learned_addr learned my addr 172.21.15.145:0/2065598484 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.828+0000 7efc3bfff640 10 client.?.objecter ms_handle_connect 0x7efc540ad6c0 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.828+0000 7efc3bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783899610 0 0) 0x7efc541e7f50 con 0x7efc540ad6c0 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc40004eb0 con 0x7efc540ad6c0 2024-03-29T11:40:23.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1903601199 0 0) 0x7efc40004eb0 con 0x7efc540ad6c0 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc541e7f50 con 0x7efc540ad6c0 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efc4c001930 con 0x7efc540ad6c0 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 10 monclient(hunting): dump: 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:23.833 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:23.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:23.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1932505510 0 0) 0x7efc541e7f50 con 0x7efc540ad6c0 2024-03-29T11:40:23.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.829+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 >> v1:172.21.15.203:6789/0 conn(0x7efc540af6c0 legacy=0x7efc541e42d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:23.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 >> v1:172.21.15.145:6789/0 conn(0x7efc5414fdc0 legacy=0x7efc5408a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:23.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:40:23.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 1 monclient: found mon.c 2024-03-29T11:40:23.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:23.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:23.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc541e80e0 con 0x7efc540ad6c0 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 10 monclient: _finish_auth 0 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc5a089640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc5a089640 10 monclient: _renew_subs 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc5a089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc541e49e0 con 0x7efc540ad6c0 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc5a089640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc5a089640 10 monclient: _renew_subs 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc5a089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5408afe0 con 0x7efc540ad6c0 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc5a089640 1 librados: init done 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc4c001ee0 con 0x7efc540ad6c0 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7efc4c001930 con 0x7efc540ad6c0 2024-03-29T11:40:23.835 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc3bfff640 10 monclient: dump: 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:23.836 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.830+0000 7efc517fa640 4 set_mon_vals no callback set 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.832+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7efc4c01ae20 con 0x7efc540ad6c0 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7efc4c054940 con 0x7efc540ad6c0 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc3bfff640 10 client.4433.objecter ms_dispatch 0x7efc54000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc3bfff640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc3bfff640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc3bfff640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc52ffd640 10 monclient: get_auth_request con 0x7efc4006dec0 auth_method 0 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc5a089640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc5a089640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc5a089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc3bfff640 10 client.4433.objecter ms_handle_connect 0x7efc4006dec0 2024-03-29T11:40:23.838 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.833+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc34005080 con 0x7efc540ad6c0 2024-03-29T11:40:23.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.847+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc4c050be0 con 0x7efc540ad6c0 2024-03-29T11:40:23.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.847+0000 7efc3bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:23.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:23.847+0000 7efc3bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:24.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.162+0000 7efc5a089640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:24.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.162+0000 7efc5a089640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:24.165 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.162+0000 7efc5a089640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:24.165 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.162+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc34005330 con 0x7efc540ad6c0 2024-03-29T11:40:24.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.165+0000 7efc3bfff640 1 -- 172.21.15.145:0/2065598484 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7efc4c050be0 con 0x7efc540ad6c0 2024-03-29T11:40:24.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.165+0000 7efc3bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:24.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.165+0000 7efc3bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:24.167 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:40:24.168 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:24.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.168+0000 7efc5a089640 10 librados: watch_flush enter 2024-03-29T11:40:24.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.168+0000 7efc5a089640 10 librados: watch_flush exit 2024-03-29T11:40:24.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.168+0000 7efc5a089640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-29T11:40:24.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.168+0000 7efc5a089640 10 client.4433.objecter successfully canceled tick 2024-03-29T11:40:24.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.169+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 >> v1:172.21.15.145:6816/227372239 conn(0x7efc4006dec0 legacy=0x7efc40070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:24.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.169+0000 7efc5a089640 10 monclient: shutdown 2024-03-29T11:40:24.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.169+0000 7efc5a089640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:24.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.169+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 >> v1:172.21.15.145:6790/0 conn(0x7efc540ad6c0 legacy=0x7efc541e7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:24.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.169+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 shutdown_connections 2024-03-29T11:40:24.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.169+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 >> 172.21.15.145:0/2065598484 conn(0x7efc540b35e0 msgr2=0x7efc540adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:24.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.169+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 shutdown_connections 2024-03-29T11:40:24.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.169+0000 7efc5a089640 1 -- 172.21.15.145:0/2065598484 wait complete. 2024-03-29T11:40:24.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:24.169+0000 7efc5a089640 1 librados: shutdown 2024-03-29T11:40:24.188 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:30.995 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:31.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.127+0000 7f6e9f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.127+0000 7f6e9f59e640 10 monclient: build_initial_monmap 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.127+0000 7f6e9f59e640 10 monclient: monmap: 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:40:31.128722+0000 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:40:31.128722+0000 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:40:31.130 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:40:31.131 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:31.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.128+0000 7f6e9f59e640 1 Processor -- start 2024-03-29T11:40:31.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.129+0000 7f6e9f59e640 1 -- start start 2024-03-29T11:40:31.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.129+0000 7f6e9f59e640 10 monclient: init 2024-03-29T11:40:31.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.130+0000 7f6e9f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:31.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.130+0000 7f6e9f59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:31.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.130+0000 7f6e9f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f6e98006890 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:31.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.130+0000 7f6e9f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f6e98006c70 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:31.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.130+0000 7f6e9e59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6e98006890 0x7f6e980c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:41318/0 (socket says 172.21.15.145:41318) 2024-03-29T11:40:31.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.130+0000 7f6e9dd9b640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f6e98006c70 0x7f6e980c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:48568/0 (socket says 172.21.15.145:48568) 2024-03-29T11:40:31.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.130+0000 7f6e9f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f6e9802b4c0 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:31.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.131+0000 7f6e9f59e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9802c830 con 0x7f6e9802b4c0 2024-03-29T11:40:31.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.131+0000 7f6e9f59e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9802afa0 con 0x7f6e98006890 2024-03-29T11:40:31.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.131+0000 7f6e9e59c640 1 -- 172.21.15.145:0/3001860946 learned_addr learned my addr 172.21.15.145:0/3001860946 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:31.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.131+0000 7f6e9f59e640 1 -- 172.21.15.145:0/3001860946 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9802aba0 con 0x7f6e98006c70 2024-03-29T11:40:31.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.131+0000 7f6e9f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:31.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.131+0000 7f6e9f59e640 10 monclient(hunting): authenticate will time out at 1220.439175s 2024-03-29T11:40:31.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370064383 0 0) 0x7f6e9802afa0 con 0x7f6e98006890 2024-03-29T11:40:31.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:40:31.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:31.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:31.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e88003520 con 0x7f6e98006890 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1718884972 0 0) 0x7f6e9802aba0 con 0x7f6e98006c70 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e9802afa0 con 0x7f6e98006c70 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3941479233 0 0) 0x7f6e88003520 con 0x7f6e98006890 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 >> v1:172.21.15.145:6790/0 conn(0x7f6e98006c70 legacy=0x7f6e980c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:31.136 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.132+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 >> v1:172.21.15.145:6789/0 conn(0x7f6e9802b4c0 legacy=0x7f6e980cf970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:40:31.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.134+0000 7f6e9d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:40:31.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 1 monclient: found mon.noname-c 2024-03-29T11:40:31.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:31.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9802ad30 con 0x7f6e98006890 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 10 monclient: _finish_auth 0 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f6e94001690 con 0x7f6e98006890 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.135+0000 7f6e9d59a640 10 monclient: dump: 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:31.138 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:31.139 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.139 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.139 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:31.139 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:31.139 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:31.139 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:31.139 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:31.139 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:31.146 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9f59e640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:40:31.146 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:31.146 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e940017f0 con 0x7f6e98006890 2024-03-29T11:40:31.146 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:31.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:31.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9f59e640 4 set_mon_vals no callback set 2024-03-29T11:40:31.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9d59a640 1 -- 172.21.15.145:0/3001860946 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f6e94001bc0 con 0x7f6e98006890 2024-03-29T11:40:31.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:31.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:31.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.136+0000 7f6e9d59a640 10 monclient: dump: 2024-03-29T11:40:31.148 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:31.148 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:31.148 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.148 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.148 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:31.148 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:31.149 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.137+0000 7f6e9f59e640 10 monclient: shutdown 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.137+0000 7f6e9f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.137+0000 7f6e9f59e640 1 -- 172.21.15.145:0/3001860946 >> v1:172.21.15.203:6789/0 conn(0x7f6e98006890 legacy=0x7f6e980c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.138+0000 7f6e9f59e640 1 -- 172.21.15.145:0/3001860946 shutdown_connections 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.138+0000 7f6e9f59e640 1 -- 172.21.15.145:0/3001860946 >> 172.21.15.145:0/3001860946 conn(0x7f6e980058f0 msgr2=0x7f6e98005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.138+0000 7f6e9f59e640 1 -- 172.21.15.145:0/3001860946 shutdown_connections 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.138+0000 7f6e9f59e640 1 -- 172.21.15.145:0/3001860946 wait complete. 2024-03-29T11:40:31.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.139+0000 7f6e9f59e640 10 monclient: build_initial_monmap 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.139+0000 7f6e9f59e640 10 monclient: monmap: 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.140+0000 7f6e9f59e640 1 librados: starting msgr at 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.140+0000 7f6e9f59e640 1 librados: starting objecter 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.140+0000 7f6e9f59e640 1 Processor -- start 2024-03-29T11:40:31.151 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.140+0000 7f6e9f59e640 1 -- start start 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.140+0000 7f6e9f59e640 1 librados: setting wanted keys 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.141+0000 7f6e9f59e640 1 librados: calling monclient init 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.141+0000 7f6e9f59e640 10 monclient: init 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.141+0000 7f6e9f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.141+0000 7f6e9f59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f6e98006890 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f6e9802b4c0 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9dd9b640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f6e9802b4c0 0x7f6e980b7ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:33862/0 (socket says 172.21.15.145:33862) 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9dd9b640 1 -- 172.21.15.145:0/1521583938 learned_addr learned my addr 172.21.15.145:0/1521583938 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f6e980c15c0 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980c3270 con 0x7f6e98006890 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980c1f70 con 0x7f6e9802b4c0 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e980c1a50 con 0x7f6e980c15c0 2024-03-29T11:40:31.152 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.142+0000 7f6e9f59e640 10 monclient(hunting): authenticate will time out at 1220.450948s 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 client.?.objecter ms_handle_connect 0x7f6e980c15c0 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 client.?.objecter ms_handle_connect 0x7f6e9802b4c0 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216850466 0 0) 0x7f6e980c1a50 con 0x7f6e980c15c0 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e80004dd0 con 0x7f6e980c15c0 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972777209 0 0) 0x7f6e980c1f70 con 0x7f6e9802b4c0 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:31.153 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:31.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:40:31.154 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.143+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e980c1a50 con 0x7f6e9802b4c0 2024-03-29T11:40:31.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1802718260 0 0) 0x7f6e80004dd0 con 0x7f6e980c15c0 2024-03-29T11:40:31.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e980c1f70 con 0x7f6e980c15c0 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f6ea004fdd0 con 0x7f6e980c15c0 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 10 monclient(hunting): dump: 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:31.157 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4063987061 0 0) 0x7f6e980c1a50 con 0x7f6e9802b4c0 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e80004dd0 con 0x7f6e9802b4c0 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f6e90001930 con 0x7f6e9802b4c0 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 10 monclient(hunting): dump: 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:31.158 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1892019388 0 0) 0x7f6e980c1f70 con 0x7f6e980c15c0 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 >> v1:172.21.15.203:6789/0 conn(0x7f6e9802b4c0 legacy=0x7f6e980b7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.144+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 >> v1:172.21.15.145:6789/0 conn(0x7f6e98006890 legacy=0x7f6e98158d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e8effd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e8effd640 1 monclient: found mon.c 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e8effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e980bd290 con 0x7f6e980c15c0 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e8effd640 10 monclient: _finish_auth 0 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e8effd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:31.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e9f59e640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e9f59e640 10 monclient: _renew_subs 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e9f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e980bd480 con 0x7f6e980c15c0 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e9f59e640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e9f59e640 10 monclient: _renew_subs 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e9f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e98159670 con 0x7f6e980c15c0 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.145+0000 7f6e9f59e640 1 librados: init done 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.146+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ea004f400 con 0x7f6e980c15c0 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.146+0000 7f6e8effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.146+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f6ea0075710 con 0x7f6e980c15c0 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.146+0000 7f6e8effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.146+0000 7f6e8effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:31.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.146+0000 7f6e8effd640 10 monclient: dump: 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.146+0000 7f6e9cd99640 4 set_mon_vals no callback set 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.147+0000 7f6e9f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.148+0000 7f6e9f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:31.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.148+0000 7f6e9f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.148+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e84005080 con 0x7f6e980c15c0 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.148+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f6ea0067bf0 con 0x7f6e980c15c0 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.149+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f6ea00fe2c0 con 0x7f6e980c15c0 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.149+0000 7f6e8effd640 10 client.4439.objecter ms_dispatch 0x7f6ea0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.149+0000 7f6e8effd640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.149+0000 7f6e8effd640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.149+0000 7f6e8effd640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.149+0000 7f6e9e59c640 10 monclient: get_auth_request con 0x7f6e80072080 auth_method 0 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.149+0000 7f6e8effd640 10 client.4439.objecter ms_handle_connect 0x7f6e80072080 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.154+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ea00c9ba0 con 0x7f6e980c15c0 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.154+0000 7f6e8effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:31.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.154+0000 7f6e8effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:31.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.477+0000 7f6e9f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:31.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.477+0000 7f6e9f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:31.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.477+0000 7f6e9f59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:31.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.477+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e84005330 con 0x7f6e980c15c0 2024-03-29T11:40:31.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.478+0000 7f6e8effd640 1 -- 172.21.15.145:0/1521583938 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f6ea00cbad0 con 0x7f6e980c15c0 2024-03-29T11:40:31.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.478+0000 7f6e8effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:31.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.478+0000 7f6e8effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:31.483 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:40:31.483 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:31.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.482+0000 7f6e9f59e640 10 librados: watch_flush enter 2024-03-29T11:40:31.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.482+0000 7f6e9f59e640 10 librados: watch_flush exit 2024-03-29T11:40:31.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.482+0000 7f6e9f59e640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:40:31.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.482+0000 7f6e9f59e640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:40:31.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.483+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 >> v1:172.21.15.145:6816/227372239 conn(0x7f6e80072080 legacy=0x7f6e80074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:31.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.483+0000 7f6e9f59e640 10 monclient: shutdown 2024-03-29T11:40:31.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.483+0000 7f6e9f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:31.485 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.483+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 >> v1:172.21.15.145:6790/0 conn(0x7f6e980c15c0 legacy=0x7f6e98164680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:31.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.486+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 shutdown_connections 2024-03-29T11:40:31.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.486+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 >> 172.21.15.145:0/1521583938 conn(0x7f6e980058f0 msgr2=0x7f6e98038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:31.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.486+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 shutdown_connections 2024-03-29T11:40:31.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.487+0000 7f6e9f59e640 1 -- 172.21.15.145:0/1521583938 wait complete. 2024-03-29T11:40:31.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:31.487+0000 7f6e9f59e640 1 librados: shutdown 2024-03-29T11:40:31.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:38.316 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:38.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.467+0000 7f36f6a2a640 10 monclient: get_monmap_and_config 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.467+0000 7f36f6a2a640 10 monclient: build_initial_monmap 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.467+0000 7f36f6a2a640 10 monclient: monmap: 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:40:38.468685+0000 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:40:38.468685+0000 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:40:38.470 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:38.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.468+0000 7f36f6a2a640 1 Processor -- start 2024-03-29T11:40:38.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.468+0000 7f36f6a2a640 1 -- start start 2024-03-29T11:40:38.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.468+0000 7f36f6a2a640 10 monclient: init 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f36e8006810 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 10 monclient(hunting): picked mon.noname-b con 0x7f36e8006bf0 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f36e80c78f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e80c7f30 con 0x7f36e8006810 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e8036fd0 con 0x7f36e80c78f0 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e80c54f0 con 0x7f36e8006bf0 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:38.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.469+0000 7f36f6a2a640 10 monclient(hunting): authenticate will time out at 1227.777842s 2024-03-29T11:40:38.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.470+0000 7f36effff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f36e8006810 0x7f36e8033620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:49380/0 (socket says 172.21.15.145:49380) 2024-03-29T11:40:38.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.470+0000 7f36effff640 1 -- 172.21.15.145:0/79053373 learned_addr learned my addr 172.21.15.145:0/79053373 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:38.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313916922 0 0) 0x7f36e80c7f30 con 0x7f36e8006810 2024-03-29T11:40:38.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:40:38.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e0003520 con 0x7f36e8006810 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831080750 0 0) 0x7f36e8036fd0 con 0x7f36e80c78f0 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e80c7f30 con 0x7f36e80c78f0 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1093548111 0 0) 0x7f36e80c7f30 con 0x7f36e80c78f0 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 >> v1:172.21.15.145:6790/0 conn(0x7f36e8006bf0 legacy=0x7f36e80cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 >> v1:172.21.15.145:6789/0 conn(0x7f36e8006810 legacy=0x7f36e8033620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 1 monclient: found mon.noname-c 2024-03-29T11:40:38.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.471+0000 7f36eeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e8037240 con 0x7f36e80c78f0 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36f6a2a640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36f6a2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f36e4001670 con 0x7f36e80c78f0 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 10 monclient: dump: 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:38.477 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36f6a2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36e4001890 con 0x7f36e80c78f0 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:38.478 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 1 -- 172.21.15.145:0/79053373 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f36e4001ba0 con 0x7f36e80c78f0 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.472+0000 7f36eeffd640 10 monclient: dump: 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.473+0000 7f36f6a2a640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:38.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.473+0000 7f36f6a2a640 4 set_mon_vals no callback set 2024-03-29T11:40:38.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.474+0000 7f36f6a2a640 10 monclient: shutdown 2024-03-29T11:40:38.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.474+0000 7f36f6a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:38.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.474+0000 7f36f6a2a640 1 -- 172.21.15.145:0/79053373 >> v1:172.21.15.203:6789/0 conn(0x7f36e80c78f0 legacy=0x7f36e80d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.474+0000 7f36f6a2a640 1 -- 172.21.15.145:0/79053373 shutdown_connections 2024-03-29T11:40:38.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.474+0000 7f36f6a2a640 1 -- 172.21.15.145:0/79053373 >> 172.21.15.145:0/79053373 conn(0x7f36e8005c40 msgr2=0x7f36e8006050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:38.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.474+0000 7f36f6a2a640 1 -- 172.21.15.145:0/79053373 shutdown_connections 2024-03-29T11:40:38.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.474+0000 7f36f6a2a640 1 -- 172.21.15.145:0/79053373 wait complete. 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.475+0000 7f36f6a2a640 10 monclient: build_initial_monmap 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.475+0000 7f36f6a2a640 10 monclient: monmap: 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.475+0000 7f36f6a2a640 1 librados: starting msgr at 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.475+0000 7f36f6a2a640 1 librados: starting objecter 2024-03-29T11:40:38.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.475+0000 7f36f6a2a640 1 Processor -- start 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.476+0000 7f36f6a2a640 1 -- start start 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.476+0000 7f36f6a2a640 1 librados: setting wanted keys 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.476+0000 7f36f6a2a640 1 librados: calling monclient init 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.476+0000 7f36f6a2a640 10 monclient: init 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 10 monclient(hunting): picked mon.noname-a con 0x7f36e8006810 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 10 monclient(hunting): picked mon.noname-c con 0x7f36e80c78f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 10 monclient(hunting): picked mon.noname-b con 0x7f36e80c9410 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36effff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f36e8006810 0x7f36e80c8d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:49558/0 (socket says 172.21.15.145:49558) 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36effff640 1 -- 172.21.15.145:0/1809549629 learned_addr learned my addr 172.21.15.145:0/1809549629 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e802f8c0 con 0x7f36e8006810 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e8160f20 con 0x7f36e80c9410 2024-03-29T11:40:38.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e80c0be0 con 0x7f36e80c78f0 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36f6a2a640 10 monclient(hunting): authenticate will time out at 1227.785470s 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36ecff9640 10 client.?.objecter ms_handle_connect 0x7f36e8006810 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36ecff9640 10 client.?.objecter ms_handle_connect 0x7f36e80c78f0 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.477+0000 7f36ecff9640 10 client.?.objecter ms_handle_connect 0x7f36e80c9410 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950061194 0 0) 0x7f36e80c0be0 con 0x7f36e80c78f0 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36d4004eb0 con 0x7f36e80c78f0 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544144729 0 0) 0x7f36e802f8c0 con 0x7f36e8006810 2024-03-29T11:40:38.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e80c0be0 con 0x7f36e8006810 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676785449 0 0) 0x7f36e8160f20 con 0x7f36e80c9410 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.478+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36e802f8c0 con 0x7f36e80c9410 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2425360892 0 0) 0x7f36e80c0be0 con 0x7f36e8006810 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e8160f20 con 0x7f36e8006810 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2312182647 0 0) 0x7f36e802f8c0 con 0x7f36e80c9410 2024-03-29T11:40:38.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e80c0be0 con 0x7f36e80c9410 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f36dc001a70 con 0x7f36e8006810 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): dump: 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:38.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f36e4001530 con 0x7f36e80c9410 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): dump: 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:38.494 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871076833 0 0) 0x7f36d4004eb0 con 0x7f36e80c78f0 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36e802f8c0 con 0x7f36e80c78f0 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f36f004fdd0 con 0x7f36e80c78f0 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.479+0000 7f36ecff9640 10 monclient(hunting): dump: 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:38.495 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3259936959 0 0) 0x7f36e802f8c0 con 0x7f36e80c78f0 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 >> v1:172.21.15.203:6789/0 conn(0x7f36e80c9410 legacy=0x7f36e8161440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 >> v1:172.21.15.145:6789/0 conn(0x7f36e8006810 legacy=0x7f36e80c8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 1 monclient: found mon.c 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e80300c0 con 0x7f36e80c78f0 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36f6a2a640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36f6a2a640 10 monclient: _renew_subs 2024-03-29T11:40:38.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36f6a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e80cac70 con 0x7f36e80c78f0 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36f6a2a640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36f6a2a640 10 monclient: _renew_subs 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36f6a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e80cbce0 con 0x7f36e80c78f0 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.480+0000 7f36f6a2a640 1 librados: init done 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.481+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36f004f400 con 0x7f36e80c78f0 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.481+0000 7f36ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.481+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f36f0075710 con 0x7f36e80c78f0 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.481+0000 7f36ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.481+0000 7f36ecff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.481+0000 7f36ee7fc640 4 set_mon_vals no callback set 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.481+0000 7f36ecff9640 10 monclient: dump: 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:38.497 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.482+0000 7f36f6a2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.482+0000 7f36f6a2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.482+0000 7f36f6a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.482+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d0005080 con 0x7f36e80c78f0 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.483+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f36f0067bf0 con 0x7f36e80c78f0 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.484+0000 7f36effff640 10 monclient: get_auth_request con 0x7f36d406de20 auth_method 0 2024-03-29T11:40:38.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.485+0000 7f36ecff9640 10 client.4445.objecter ms_handle_connect 0x7f36d406de20 2024-03-29T11:40:38.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.485+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f36f00fe300 con 0x7f36e80c78f0 2024-03-29T11:40:38.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.485+0000 7f36ecff9640 10 client.4445.objecter ms_dispatch 0x7f36f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:38.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.485+0000 7f36ecff9640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:38.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.485+0000 7f36ecff9640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:38.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.485+0000 7f36ecff9640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:40:38.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.494+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36f00c9be0 con 0x7f36e80c78f0 2024-03-29T11:40:38.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.494+0000 7f36ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:38.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.494+0000 7f36ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:38.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.875+0000 7f36f6a2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:38.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.875+0000 7f36f6a2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:38.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.875+0000 7f36f6a2a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:38.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.875+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36d0005330 con 0x7f36e80c78f0 2024-03-29T11:40:38.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.877+0000 7f36ecff9640 1 -- 172.21.15.145:0/1809549629 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f36f0067f20 con 0x7f36e80c78f0 2024-03-29T11:40:38.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.877+0000 7f36ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:38.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.877+0000 7f36ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:38.880 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:40:38.880 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:38.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.881+0000 7f36f6a2a640 10 librados: watch_flush enter 2024-03-29T11:40:38.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.881+0000 7f36f6a2a640 10 librados: watch_flush exit 2024-03-29T11:40:38.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.881+0000 7f36f6a2a640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:40:38.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.881+0000 7f36f6a2a640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:40:38.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.881+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 >> v1:172.21.15.145:6816/227372239 conn(0x7f36d406de20 legacy=0x7f36d40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.881+0000 7f36f6a2a640 10 monclient: shutdown 2024-03-29T11:40:38.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.881+0000 7f36f6a2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:38.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.881+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 >> v1:172.21.15.145:6790/0 conn(0x7f36e80c78f0 legacy=0x7f36e80cb3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:38.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.882+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 shutdown_connections 2024-03-29T11:40:38.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.882+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 >> 172.21.15.145:0/1809549629 conn(0x7f36e8005c40 msgr2=0x7f36e8005110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:38.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.882+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 shutdown_connections 2024-03-29T11:40:38.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.882+0000 7f36f6a2a640 1 -- 172.21.15.145:0/1809549629 wait complete. 2024-03-29T11:40:38.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:38.882+0000 7f36f6a2a640 1 librados: shutdown 2024-03-29T11:40:38.901 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:45.708 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:45.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.855+0000 7fef2e525640 10 monclient: get_monmap_and_config 2024-03-29T11:40:45.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.855+0000 7fef2e525640 10 monclient: build_initial_monmap 2024-03-29T11:40:45.857 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.855+0000 7fef2e525640 10 monclient: monmap: 2024-03-29T11:40:45.857 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:45.857 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:45.858 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:40:45.856227+0000 2024-03-29T11:40:45.858 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:40:45.856227+0000 2024-03-29T11:40:45.858 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:45.858 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:45.858 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:45.858 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:40:45.858 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:40:45.858 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:45.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.857+0000 7fef2e525640 1 Processor -- start 2024-03-29T11:40:45.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.857+0000 7fef2e525640 1 -- start start 2024-03-29T11:40:45.859 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.857+0000 7fef2e525640 10 monclient: init 2024-03-29T11:40:45.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:45.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:45.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 10 monclient(hunting): picked mon.noname-a con 0x7fef2806f050 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:45.860 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 10 monclient(hunting): picked mon.noname-c con 0x7fef28070240 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 10 monclient(hunting): picked mon.noname-b con 0x7fef2806f430 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2813f400 con 0x7fef2806f050 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef281429c0 con 0x7fef28070240 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2806ff80 con 0x7fef2806f430 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2cac2640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fef2806f430 0x7fef2814a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:42380/0 (socket says 172.21.15.145:42380) 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2cac2640 1 -- 172.21.15.145:0/4132752196 learned_addr learned my addr 172.21.15.145:0/4132752196 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.858+0000 7fef2e525640 10 monclient(hunting): authenticate will time out at 1235.165959s 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2792415004 0 0) 0x7fef2813f400 con 0x7fef2806f050 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:40:45.861 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef10003520 con 0x7fef2806f050 2024-03-29T11:40:45.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313719983 0 0) 0x7fef2806ff80 con 0x7fef2806f430 2024-03-29T11:40:45.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:45.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:45.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:45.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:40:45.862 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef2813f400 con 0x7fef2806f430 2024-03-29T11:40:45.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2981094214 0 0) 0x7fef2813f400 con 0x7fef2806f430 2024-03-29T11:40:45.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 >> v1:172.21.15.203:6789/0 conn(0x7fef28070240 legacy=0x7fef280ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:45.864 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 >> v1:172.21.15.145:6789/0 conn(0x7fef2806f050 legacy=0x7fef281422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 1 monclient: found mon.noname-b 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.859+0000 7fef26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef280aef70 con 0x7fef2806f430 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fef18001670 con 0x7fef2806f430 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: dump: 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:45.865 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef2e525640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef2e525640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef18001890 con 0x7fef2806f430 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 1 -- 172.21.15.145:0/4132752196 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fef18001c20 con 0x7fef2806f430 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef2e525640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:45.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef2e525640 4 set_mon_vals no callback set 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.860+0000 7fef26ffd640 10 monclient: dump: 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.861+0000 7fef2e525640 10 monclient: shutdown 2024-03-29T11:40:45.867 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.861+0000 7fef2e525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.861+0000 7fef2e525640 1 -- 172.21.15.145:0/4132752196 >> v1:172.21.15.145:6790/0 conn(0x7fef2806f430 legacy=0x7fef2814a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.861+0000 7fef2e525640 1 -- 172.21.15.145:0/4132752196 shutdown_connections 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.861+0000 7fef2e525640 1 -- 172.21.15.145:0/4132752196 >> 172.21.15.145:0/4132752196 conn(0x7fef280b4930 msgr2=0x7fef280b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.861+0000 7fef2e525640 1 -- 172.21.15.145:0/4132752196 shutdown_connections 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.861+0000 7fef2e525640 1 -- 172.21.15.145:0/4132752196 wait complete. 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.863+0000 7fef2e525640 10 monclient: build_initial_monmap 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.863+0000 7fef2e525640 10 monclient: monmap: 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:45.868 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:40:45.869 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:40:45.869 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:45.872 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.863+0000 7fef2e525640 1 librados: starting msgr at 2024-03-29T11:40:45.872 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.863+0000 7fef2e525640 1 librados: starting objecter 2024-03-29T11:40:45.872 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.863+0000 7fef2e525640 1 Processor -- start 2024-03-29T11:40:45.872 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.864+0000 7fef2e525640 1 -- start start 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.864+0000 7fef2e525640 1 librados: setting wanted keys 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.864+0000 7fef2e525640 1 librados: calling monclient init 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.864+0000 7fef2e525640 10 monclient: init 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 10 monclient(hunting): picked mon.noname-a con 0x7fef2806f050 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 10 monclient(hunting): picked mon.noname-b con 0x7fef2806f430 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 10 monclient(hunting): picked mon.noname-c con 0x7fef28070240 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef280ad330 con 0x7fef2806f050 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef280ad560 con 0x7fef2806f430 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef280ae390 con 0x7fef28070240 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef2e525640 10 monclient(hunting): authenticate will time out at 1235.172363s 2024-03-29T11:40:45.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef27fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fef2806f050 0x7fef280acc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:42128/0 (socket says 172.21.15.145:42128) 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef27fff640 1 -- 172.21.15.145:0/1119109471 learned_addr learned my addr 172.21.15.145:0/1119109471 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 client.?.objecter ms_handle_connect 0x7fef2806f050 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 client.?.objecter ms_handle_connect 0x7fef28070240 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486922617 0 0) 0x7fef280ad330 con 0x7fef2806f050 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef14004eb0 con 0x7fef2806f050 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 client.?.objecter ms_handle_connect 0x7fef2806f430 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.865+0000 7fef24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:45.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293024949 0 0) 0x7fef280ae390 con 0x7fef28070240 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef140051d0 con 0x7fef28070240 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1166448034 0 0) 0x7fef14004eb0 con 0x7fef2806f050 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef280ae390 con 0x7fef2806f050 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fef20001d50 con 0x7fef2806f050 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:45.875 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.876 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.876 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.876 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:45.876 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:45.876 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:45.876 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.866+0000 7fef24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3012919786 0 0) 0x7fef140051d0 con 0x7fef28070240 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef14005b10 con 0x7fef28070240 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fef180015b0 con 0x7fef28070240 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.877 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 888917575 0 0) 0x7fef280ae390 con 0x7fef2806f050 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 >> v1:172.21.15.145:6790/0 conn(0x7fef28070240 legacy=0x7fef281d7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 >> v1:172.21.15.203:6789/0 conn(0x7fef2806f430 legacy=0x7fef28136a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 1 monclient: found mon.a 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef280ad8f0 con 0x7fef2806f050 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:45.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef2e525640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef2e525640 10 monclient: _renew_subs 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef2e525640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef280adb60 con 0x7fef2806f050 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef2e525640 10 client.4431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef2e525640 10 monclient: _renew_subs 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef2e525640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef28137360 con 0x7fef2806f050 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.867+0000 7fef2e525640 1 librados: init done 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.868+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef20003970 con 0x7fef2806f050 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.868+0000 7fef24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.868+0000 7fef25ffb640 4 set_mon_vals no callback set 2024-03-29T11:40:45.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.868+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fef20001d50 con 0x7fef2806f050 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.868+0000 7fef24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.868+0000 7fef24ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.868+0000 7fef24ff9640 10 monclient: dump: 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.870+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fef20003c20 con 0x7fef2806f050 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.870+0000 7fef2e525640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:45.880 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.870+0000 7fef2e525640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:45.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.870+0000 7fef2e525640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:45.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.870+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef08005080 con 0x7fef2806f050 2024-03-29T11:40:45.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.871+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fef20084890 con 0x7fef2806f050 2024-03-29T11:40:45.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.871+0000 7fef24ff9640 10 client.4431.objecter ms_dispatch 0x7fef28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:45.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.871+0000 7fef24ff9640 3 client.4431.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:45.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.871+0000 7fef24ff9640 3 client.4431.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:45.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.871+0000 7fef24ff9640 20 client.4431.objecter dump_active .. 0 homeless 2024-03-29T11:40:45.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.871+0000 7fef277fe640 10 monclient: get_auth_request con 0x7fef1406e560 auth_method 0 2024-03-29T11:40:45.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.872+0000 7fef24ff9640 10 client.4431.objecter ms_handle_connect 0x7fef1406e560 2024-03-29T11:40:45.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.879+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef200500e0 con 0x7fef2806f050 2024-03-29T11:40:45.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.879+0000 7fef24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:45.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:45.879+0000 7fef24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:46.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.198+0000 7fef2e525640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:46.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.198+0000 7fef2e525640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:46.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.198+0000 7fef2e525640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:40:46.199 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.198+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef08005330 con 0x7fef2806f050 2024-03-29T11:40:46.200 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:40:46.200 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:46.200 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.198+0000 7fef24ff9640 1 -- 172.21.15.145:0/1119109471 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fef200500e0 con 0x7fef2806f050 2024-03-29T11:40:46.200 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.198+0000 7fef24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:46.200 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.198+0000 7fef24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 10 librados: watch_flush enter 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 10 librados: watch_flush exit 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 20 client.4431.objecter shutdown clearing up homeless session... 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 10 client.4431.objecter successfully canceled tick 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 >> v1:172.21.15.145:6816/227372239 conn(0x7fef1406e560 legacy=0x7fef14070a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 10 monclient: shutdown 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 >> v1:172.21.15.145:6789/0 conn(0x7fef2806f050 legacy=0x7fef280acc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 shutdown_connections 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 >> 172.21.15.145:0/1119109471 conn(0x7fef280b4930 msgr2=0x7fef280b1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.202+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 shutdown_connections 2024-03-29T11:40:46.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.203+0000 7fef2e525640 1 -- 172.21.15.145:0/1119109471 wait complete. 2024-03-29T11:40:46.205 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:46.203+0000 7fef2e525640 1 librados: shutdown 2024-03-29T11:40:46.218 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:40:53.025 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:40:53.172 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.169+0000 7f10cd146640 10 monclient: get_monmap_and_config 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.169+0000 7f10cd146640 10 monclient: build_initial_monmap 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.169+0000 7f10cd146640 10 monclient: monmap: 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:40:53.170787+0000 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:40:53.170787+0000 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:40:53.173 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:53.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.170+0000 7f10cd146640 1 Processor -- start 2024-03-29T11:40:53.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 1 -- start start 2024-03-29T11:40:53.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 10 monclient: init 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 10 monclient(hunting): picked mon.noname-c con 0x7f10c80b3f30 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 10 monclient(hunting): picked mon.noname-b con 0x7f10c813b9d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 10 monclient(hunting): picked mon.noname-a con 0x7f10c80a9750 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c80ac130 con 0x7f10c80a9750 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c80ac410 con 0x7f10c80b3f30 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c80afa40 con 0x7f10c813b9d0 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:53.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.171+0000 7f10cd146640 10 monclient(hunting): authenticate will time out at 1242.479949s 2024-03-29T11:40:53.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.175+0000 7f10cc945640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f10c80a9750 0x7f10c814a040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:45423/0 (socket says 172.21.15.145:45423) 2024-03-29T11:40:53.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.175+0000 7f10cc945640 1 -- 172.21.15.145:0/2603256959 learned_addr learned my addr 172.21.15.145:0/2603256959 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:53.178 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.176+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751754315 0 0) 0x7f10c80ac410 con 0x7f10c80b3f30 2024-03-29T11:40:53.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.176+0000 7f10c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:53.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.176+0000 7f10c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:53.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.176+0000 7f10c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:53.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.176+0000 7f10c6ffd640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:40:53.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.176+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b0005730 con 0x7f10c80b3f30 2024-03-29T11:40:53.179 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706874795 0 0) 0x7f10c80ac130 con 0x7f10c80a9750 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10c80ac410 con 0x7f10c80a9750 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 12543242 0 0) 0x7f10b0005730 con 0x7f10c80b3f30 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 >> v1:172.21.15.145:6790/0 conn(0x7f10c813b9d0 legacy=0x7f10c8146910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 >> v1:172.21.15.145:6789/0 conn(0x7f10c80a9750 legacy=0x7f10c814a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:53.180 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c80aff90 con 0x7f10c80b3f30 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f10b8001690 con 0x7f10c80b3f30 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: dump: 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:53.181 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10cd146640 5 monclient: authenticate success, global_id 4435 2024-03-29T11:40:53.182 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10cd146640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b80017f0 con 0x7f10c80b3f30 2024-03-29T11:40:53.183 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 1 -- 172.21.15.145:0/2603256959 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f10b8001c00 con 0x7f10c80b3f30 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10c6ffd640 10 monclient: dump: 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:53.184 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10cd146640 10 monclient: get_monmap_and_config success 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.177+0000 7f10cd146640 4 set_mon_vals no callback set 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.178+0000 7f10cd146640 10 monclient: shutdown 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.178+0000 7f10cd146640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.178+0000 7f10cd146640 1 -- 172.21.15.145:0/2603256959 >> v1:172.21.15.203:6789/0 conn(0x7f10c80b3f30 legacy=0x7f10c813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.179+0000 7f10cd146640 1 -- 172.21.15.145:0/2603256959 shutdown_connections 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.179+0000 7f10cd146640 1 -- 172.21.15.145:0/2603256959 >> 172.21.15.145:0/2603256959 conn(0x7f10c80673e0 msgr2=0x7f10c80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.180+0000 7f10cd146640 1 -- 172.21.15.145:0/2603256959 shutdown_connections 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.180+0000 7f10cd146640 1 -- 172.21.15.145:0/2603256959 wait complete. 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.181+0000 7f10cd146640 10 monclient: build_initial_monmap 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.181+0000 7f10cd146640 10 monclient: monmap: 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:40:53.185 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:40:53.186 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:40:53.186 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:40:53.186 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:40:53.186 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:53.186 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:40:53.186 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:40:53.186 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:40:53.186 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:53.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.182+0000 7f10cd146640 1 librados: starting msgr at 2024-03-29T11:40:53.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.182+0000 7f10cd146640 1 librados: starting objecter 2024-03-29T11:40:53.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.182+0000 7f10cd146640 1 Processor -- start 2024-03-29T11:40:53.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.182+0000 7f10cd146640 1 -- start start 2024-03-29T11:40:53.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.182+0000 7f10cd146640 1 librados: setting wanted keys 2024-03-29T11:40:53.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.182+0000 7f10cd146640 1 librados: calling monclient init 2024-03-29T11:40:53.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.182+0000 7f10cd146640 10 monclient: init 2024-03-29T11:40:53.187 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.183+0000 7f10cd146640 10 monclient: _reopen_session rank -1 2024-03-29T11:40:53.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.183+0000 7f10cd146640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:40:53.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.183+0000 7f10cd146640 10 monclient(hunting): picked mon.noname-c con 0x7f10c80a9750 addr v1:172.21.15.145:6790/0 2024-03-29T11:40:53.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.184+0000 7f10cd146640 10 monclient(hunting): picked mon.noname-b con 0x7f10c80b3f30 addr v1:172.21.15.203:6789/0 2024-03-29T11:40:53.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.184+0000 7f10cd146640 10 monclient(hunting): picked mon.noname-a con 0x7f10c813b9d0 addr v1:172.21.15.145:6789/0 2024-03-29T11:40:53.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.184+0000 7f10cd146640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c81343a0 con 0x7f10c813b9d0 2024-03-29T11:40:53.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.184+0000 7f10cd146640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c813a0a0 con 0x7f10c80b3f30 2024-03-29T11:40:53.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.184+0000 7f10cd146640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c81406e0 con 0x7f10c80a9750 2024-03-29T11:40:53.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.184+0000 7f10cd146640 10 monclient(hunting): _renew_subs 2024-03-29T11:40:53.188 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.184+0000 7f10cd146640 10 monclient(hunting): authenticate will time out at 1242.492076s 2024-03-29T11:40:53.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.185+0000 7f10c7fff640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f10c80a9750 0x7f10c81d2ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:52978/0 (socket says 172.21.15.145:52978) 2024-03-29T11:40:53.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.185+0000 7f10c7fff640 1 -- 172.21.15.145:0/2040107023 learned_addr learned my addr 172.21.15.145:0/2040107023 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:40:53.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.185+0000 7f10c4ff9640 10 client.?.objecter ms_handle_connect 0x7f10c80a9750 2024-03-29T11:40:53.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.185+0000 7f10c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:40:53.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3811870430 0 0) 0x7f10c81406e0 con 0x7f10c80a9750 2024-03-29T11:40:53.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:40:53.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:40:53.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:40:53.189 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10b4004eb0 con 0x7f10c80a9750 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2069743330 0 0) 0x7f10b4004eb0 con 0x7f10c80a9750 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10c81406e0 con 0x7f10c80a9750 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f10b8001690 con 0x7f10c80a9750 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 10 monclient(hunting): dump: 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:53.190 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:53.191 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:53.191 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:53.191 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:53.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:40:53.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 833404904 0 0) 0x7f10c81406e0 con 0x7f10c80a9750 2024-03-29T11:40:53.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.186+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 >> v1:172.21.15.203:6789/0 conn(0x7f10c80b3f30 legacy=0x7f10c8133580 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:53.191 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.188+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 >> v1:172.21.15.145:6789/0 conn(0x7f10c813b9d0 legacy=0x7f10c8133c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:40:53.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:40:53.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 1 monclient: found mon.c 2024-03-29T11:40:53.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:40:53.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:53.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c8140870 con 0x7f10c80a9750 2024-03-29T11:40:53.193 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10cd146640 5 monclient: authenticate success, global_id 4454 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10cd146640 10 monclient: _renew_subs 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10cd146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c813c0a0 con 0x7f10c80a9750 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10cd146640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10cd146640 10 monclient: _renew_subs 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10cd146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c81d3600 con 0x7f10c80a9750 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10cd146640 1 librados: init done 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10b80038a0 con 0x7f10c80a9750 2024-03-29T11:40:53.194 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f10b8003cf0 con 0x7f10c80a9750 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.190+0000 7f10c4ff9640 10 monclient: dump: 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:40:53.195 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:40:53.196 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.191+0000 7f10c5ffb640 4 set_mon_vals no callback set 2024-03-29T11:40:53.196 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.191+0000 7f10cd146640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:53.197 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.192+0000 7f10cd146640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:53.197 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.192+0000 7f10cd146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:53.197 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.192+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a8005080 con 0x7f10c80a9750 2024-03-29T11:40:53.197 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.193+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f10b8003e50 con 0x7f10c80a9750 2024-03-29T11:40:53.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.193+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f10b8084c90 con 0x7f10c80a9750 2024-03-29T11:40:53.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.193+0000 7f10c4ff9640 10 client.4454.objecter ms_dispatch 0x7f10c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:40:53.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.193+0000 7f10c4ff9640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:40:53.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.193+0000 7f10c4ff9640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:40:53.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.193+0000 7f10c4ff9640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T11:40:53.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.193+0000 7f10c77fe640 10 monclient: get_auth_request con 0x7f10b406dcc0 auth_method 0 2024-03-29T11:40:53.198 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.194+0000 7f10c4ff9640 10 client.4454.objecter ms_handle_connect 0x7f10b406dcc0 2024-03-29T11:40:53.208 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.205+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10b8050570 con 0x7f10c80a9750 2024-03-29T11:40:53.208 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.205+0000 7f10c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:40:53.208 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.205+0000 7f10c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:40:53.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.528+0000 7f10cd146640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:53.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.528+0000 7f10cd146640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:53.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.528+0000 7f10cd146640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:40:53.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.528+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10a8005330 con 0x7f10c80a9750 2024-03-29T11:40:53.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.529+0000 7f10c4ff9640 1 -- 172.21.15.145:0/2040107023 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f10b80524a0 con 0x7f10c80a9750 2024-03-29T11:40:53.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.529+0000 7f10c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:40:53.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.529+0000 7f10c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:40:53.534 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:40:53.534 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:40:53.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.533+0000 7f10cd146640 10 librados: watch_flush enter 2024-03-29T11:40:53.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.536+0000 7f10cd146640 10 librados: watch_flush exit 2024-03-29T11:40:53.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.536+0000 7f10cd146640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T11:40:53.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.536+0000 7f10cd146640 10 client.4454.objecter successfully canceled tick 2024-03-29T11:40:53.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.536+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 >> v1:172.21.15.145:6816/227372239 conn(0x7f10b406dcc0 legacy=0x7f10b4070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:53.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.536+0000 7f10cd146640 10 monclient: shutdown 2024-03-29T11:40:53.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.536+0000 7f10cd146640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:40:53.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.536+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 >> v1:172.21.15.145:6790/0 conn(0x7f10c80a9750 legacy=0x7f10c81d2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:40:53.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.538+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 shutdown_connections 2024-03-29T11:40:53.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.538+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 >> 172.21.15.145:0/2040107023 conn(0x7f10c80673e0 msgr2=0x7f10c813b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:40:53.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.538+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 shutdown_connections 2024-03-29T11:40:53.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.539+0000 7f10cd146640 1 -- 172.21.15.145:0/2040107023 wait complete. 2024-03-29T11:40:53.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:40:53.539+0000 7f10cd146640 1 librados: shutdown 2024-03-29T11:40:53.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:00.364 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.513+0000 7fdc70b32640 10 monclient: get_monmap_and_config 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.513+0000 7fdc70b32640 10 monclient: build_initial_monmap 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.513+0000 7fdc70b32640 10 monclient: monmap: 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:41:00.514914+0000 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:41:00.514914+0000 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:00.517 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.514+0000 7fdc70b32640 1 Processor -- start 2024-03-29T11:41:00.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.515+0000 7fdc70b32640 1 -- start start 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.515+0000 7fdc70b32640 10 monclient: init 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 10 monclient(hunting): picked mon.noname-a con 0x7fdc6c1422b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 10 monclient(hunting): picked mon.noname-c con 0x7fdc6c142690 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 10 monclient(hunting): picked mon.noname-b con 0x7fdc6c0b17d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c142c80 con 0x7fdc6c1422b0 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c0aae40 con 0x7fdc6c142690 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c13fbd0 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:00.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.516+0000 7fdc70b32640 10 monclient(hunting): authenticate will time out at 1249.824499s 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.518+0000 7fdc6ad9d640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fdc6c0b17d0 0x7fdc6c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:43594/0 (socket says 172.21.15.145:43594) 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.518+0000 7fdc6ad9d640 1 -- 172.21.15.145:0/2606451018 learned_addr learned my addr 172.21.15.145:0/2606451018 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.518+0000 7fdc6959a640 1 -- 172.21.15.145:0/2606451018 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810152440 0 0) 0x7fdc6c13fbd0 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.518+0000 7fdc6959a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.518+0000 7fdc6959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.518+0000 7fdc6959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.518+0000 7fdc6959a640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.518+0000 7fdc6959a640 1 -- 172.21.15.145:0/2606451018 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc54003520 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.519+0000 7fdc6959a640 1 -- 172.21.15.145:0/2606451018 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3190400267 0 0) 0x7fdc54003520 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.519+0000 7fdc6959a640 1 -- 172.21.15.145:0/2606451018 >> v1:172.21.15.203:6789/0 conn(0x7fdc6c142690 legacy=0x7fdc6c0b2590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.519+0000 7fdc6959a640 1 -- 172.21.15.145:0/2606451018 >> v1:172.21.15.145:6789/0 conn(0x7fdc6c1422b0 legacy=0x7fdc6c13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.523+0000 7fdc6959a640 10 monclient: _finish_hunting 0 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.523+0000 7fdc6959a640 1 monclient: found mon.noname-b 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.523+0000 7fdc6959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.523+0000 7fdc6959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:41:00.529 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.523+0000 7fdc6959a640 1 -- 172.21.15.145:0/2606451018 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c151940 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.523+0000 7fdc6959a640 10 monclient: _finish_auth 0 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.523+0000 7fdc6959a640 10 monclient: _check_auth_tickets 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.523+0000 7fdc6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.524+0000 7fdc70b32640 5 monclient: authenticate success, global_id 4457 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.524+0000 7fdc70b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.524+0000 7fdc6959a640 1 -- 172.21.15.145:0/2606451018 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fdc5c001690 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.524+0000 7fdc6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.524+0000 7fdc6959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.524+0000 7fdc6959a640 10 monclient: dump: 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:00.530 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:00.531 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:00.531 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:00.531 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:00.531 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:00.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.524+0000 7fdc70b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.525+0000 7fdc6959a640 1 -- 172.21.15.145:0/2606451018 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc5c0017f0 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.525+0000 7fdc6959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.525+0000 7fdc6959a640 1 -- 172.21.15.145:0/2606451018 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fdc5c001c40 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.525+0000 7fdc6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.525+0000 7fdc6959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.525+0000 7fdc6959a640 10 monclient: dump: 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:00.532 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.526+0000 7fdc70b32640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.526+0000 7fdc70b32640 4 set_mon_vals no callback set 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.527+0000 7fdc70b32640 10 monclient: shutdown 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.527+0000 7fdc70b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.527+0000 7fdc70b32640 1 -- 172.21.15.145:0/2606451018 >> v1:172.21.15.145:6790/0 conn(0x7fdc6c0b17d0 legacy=0x7fdc6c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.527+0000 7fdc70b32640 1 -- 172.21.15.145:0/2606451018 shutdown_connections 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.527+0000 7fdc70b32640 1 -- 172.21.15.145:0/2606451018 >> 172.21.15.145:0/2606451018 conn(0x7fdc6c066a00 msgr2=0x7fdc6c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.528+0000 7fdc70b32640 1 -- 172.21.15.145:0/2606451018 shutdown_connections 2024-03-29T11:41:00.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.529+0000 7fdc70b32640 1 -- 172.21.15.145:0/2606451018 wait complete. 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.531+0000 7fdc70b32640 10 monclient: build_initial_monmap 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.531+0000 7fdc70b32640 10 monclient: monmap: 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:00.534 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:41:00.535 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:41:00.535 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:00.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.531+0000 7fdc70b32640 1 librados: starting msgr at 2024-03-29T11:41:00.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.531+0000 7fdc70b32640 1 librados: starting objecter 2024-03-29T11:41:00.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.531+0000 7fdc70b32640 1 Processor -- start 2024-03-29T11:41:00.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.531+0000 7fdc70b32640 1 -- start start 2024-03-29T11:41:00.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.531+0000 7fdc70b32640 1 librados: setting wanted keys 2024-03-29T11:41:00.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.531+0000 7fdc70b32640 1 librados: calling monclient init 2024-03-29T11:41:00.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.531+0000 7fdc70b32640 10 monclient: init 2024-03-29T11:41:00.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.532+0000 7fdc70b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:00.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.532+0000 7fdc70b32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:00.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.532+0000 7fdc70b32640 10 monclient(hunting): picked mon.noname-c con 0x7fdc6c0b17d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:00.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.532+0000 7fdc70b32640 10 monclient(hunting): picked mon.noname-a con 0x7fdc6c1422b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:00.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.532+0000 7fdc70b32640 10 monclient(hunting): picked mon.noname-b con 0x7fdc6c142690 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:00.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.532+0000 7fdc70b32640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c06ebd0 con 0x7fdc6c1422b0 2024-03-29T11:41:00.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.532+0000 7fdc70b32640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c06eeb0 con 0x7fdc6c142690 2024-03-29T11:41:00.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.534+0000 7fdc70b32640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc6c131e10 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.534+0000 7fdc70b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:00.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.534+0000 7fdc6a59c640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fdc6c0b17d0 0x7fdc6c132330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:53214/0 (socket says 172.21.15.145:53214) 2024-03-29T11:41:00.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.534+0000 7fdc6a59c640 1 -- 172.21.15.145:0/3209101553 learned_addr learned my addr 172.21.15.145:0/3209101553 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:00.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.534+0000 7fdc70b32640 10 monclient(hunting): authenticate will time out at 1249.842663s 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.535+0000 7fdc5affd640 10 client.?.objecter ms_handle_connect 0x7fdc6c0b17d0 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.535+0000 7fdc5affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.535+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071040940 0 0) 0x7fdc6c131e10 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.535+0000 7fdc5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.535+0000 7fdc5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.535+0000 7fdc5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.535+0000 7fdc5affd640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.535+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc4c004eb0 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.536+0000 7fdc5affd640 10 client.?.objecter ms_handle_connect 0x7fdc6c1422b0 2024-03-29T11:41:00.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.536+0000 7fdc5affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:00.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.538+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791586508 0 0) 0x7fdc6c06ebd0 con 0x7fdc6c1422b0 2024-03-29T11:41:00.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.538+0000 7fdc5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:00.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.538+0000 7fdc5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:00.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.538+0000 7fdc5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:00.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.538+0000 7fdc5affd640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:41:00.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.538+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc6c131e10 con 0x7fdc6c1422b0 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 10 client.?.objecter ms_handle_connect 0x7fdc6c142690 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 764117979 0 0) 0x7fdc4c004eb0 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc6c06ebd0 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3886021797 0 0) 0x7fdc6c131e10 con 0x7fdc6c1422b0 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdc4c004eb0 con 0x7fdc6c1422b0 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041557 0 0) 0x7fdc6c06eeb0 con 0x7fdc6c142690 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:00.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc6c131e10 con 0x7fdc6c142690 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fdc64004400 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.539+0000 7fdc5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 10 monclient(hunting): dump: 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:00.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fdc60001710 con 0x7fdc6c1422b0 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 10 monclient(hunting): dump: 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:00.547 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1792498276 0 0) 0x7fdc6c06ebd0 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 >> v1:172.21.15.203:6789/0 conn(0x7fdc6c142690 legacy=0x7fdc6c1e6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.540+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 >> v1:172.21.15.145:6789/0 conn(0x7fdc6c1422b0 legacy=0x7fdc6c070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc5affd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc5affd640 1 monclient: found mon.c 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc5affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc6c0b6bc0 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc5affd640 10 monclient: _finish_auth 0 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc5affd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc70b32640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc70b32640 10 monclient: _renew_subs 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc70b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc70b32640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c0b6e10 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc70b32640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:00.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc70b32640 10 monclient: _renew_subs 2024-03-29T11:41:00.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc70b32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:00.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc70b32640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c070f00 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.541+0000 7fdc70b32640 1 librados: init done 2024-03-29T11:41:00.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.542+0000 7fdc58ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:00.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.542+0000 7fdc58ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:00.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.542+0000 7fdc58ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:00.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.542+0000 7fdc58ff9640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc50005080 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.543+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc64004560 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.543+0000 7fdc5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:00.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.543+0000 7fdc68d99640 4 set_mon_vals no callback set 2024-03-29T11:41:00.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.543+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fdc64001500 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.543+0000 7fdc5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:00.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.543+0000 7fdc5affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.543+0000 7fdc5affd640 10 monclient: dump: 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.543+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fdc64001720 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.544+0000 7fdc69d9b640 10 monclient: get_auth_request con 0x7fdc4c06e150 auth_method 0 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.544+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fdc64084df0 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.544+0000 7fdc5affd640 10 client.4460.objecter ms_dispatch 0x7fdc6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:00.552 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.544+0000 7fdc5affd640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:00.552 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.544+0000 7fdc5affd640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:00.552 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.544+0000 7fdc5affd640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-29T11:41:00.552 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.545+0000 7fdc5affd640 10 client.4460.objecter ms_handle_connect 0x7fdc4c06e150 2024-03-29T11:41:00.557 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.554+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdc64050750 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.557 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.554+0000 7fdc5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:00.557 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.554+0000 7fdc5affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:00.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.888+0000 7fdc58ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:00.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.888+0000 7fdc58ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:00.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.888+0000 7fdc58ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:00.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.888+0000 7fdc58ff9640 1 -- 172.21.15.145:0/3209101553 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc50005330 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.893 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:41:00.893 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:00.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.890+0000 7fdc5affd640 1 -- 172.21.15.145:0/3209101553 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fdc64052680 con 0x7fdc6c0b17d0 2024-03-29T11:41:00.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.890+0000 7fdc5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:00.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.890+0000 7fdc5affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:00.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 10 librados: watch_flush enter 2024-03-29T11:41:00.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 10 librados: watch_flush exit 2024-03-29T11:41:00.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-29T11:41:00.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 10 client.4460.objecter successfully canceled tick 2024-03-29T11:41:00.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 1 -- 172.21.15.145:0/3209101553 >> v1:172.21.15.145:6816/227372239 conn(0x7fdc4c06e150 legacy=0x7fdc4c070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 10 monclient: shutdown 2024-03-29T11:41:00.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:00.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 1 -- 172.21.15.145:0/3209101553 >> v1:172.21.15.145:6790/0 conn(0x7fdc6c0b17d0 legacy=0x7fdc6c132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:00.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 1 -- 172.21.15.145:0/3209101553 shutdown_connections 2024-03-29T11:41:00.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.894+0000 7fdc58ff9640 1 -- 172.21.15.145:0/3209101553 >> 172.21.15.145:0/3209101553 conn(0x7fdc6c066a00 msgr2=0x7fdc6c13c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:00.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.895+0000 7fdc58ff9640 1 -- 172.21.15.145:0/3209101553 shutdown_connections 2024-03-29T11:41:00.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.896+0000 7fdc58ff9640 1 -- 172.21.15.145:0/3209101553 wait complete. 2024-03-29T11:41:00.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:00.897+0000 7fdc58ff9640 1 librados: shutdown 2024-03-29T11:41:00.917 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:07.724 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:07.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.867+0000 7fe17d0ce640 10 monclient: get_monmap_and_config 2024-03-29T11:41:07.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.867+0000 7fe17d0ce640 10 monclient: build_initial_monmap 2024-03-29T11:41:07.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.867+0000 7fe17d0ce640 10 monclient: monmap: 2024-03-29T11:41:07.873 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:07.874 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:07.874 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:41:07.868413+0000 2024-03-29T11:41:07.874 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:41:07.868413+0000 2024-03-29T11:41:07.874 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:07.874 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:07.874 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:07.875 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:41:07.875 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:41:07.875 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:07.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.868+0000 7fe17d0ce640 1 Processor -- start 2024-03-29T11:41:07.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.868+0000 7fe17d0ce640 1 -- start start 2024-03-29T11:41:07.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.868+0000 7fe17d0ce640 10 monclient: init 2024-03-29T11:41:07.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:07.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:07.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 10 monclient(hunting): picked mon.noname-c con 0x7fe1780b4100 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:07.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 10 monclient(hunting): picked mon.noname-a con 0x7fe17813ee90 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:07.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 10 monclient(hunting): picked mon.noname-b con 0x7fe1780a9ac0 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:07.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1780af6f0 con 0x7fe17813ee90 2024-03-29T11:41:07.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1780aac80 con 0x7fe1780b4100 2024-03-29T11:41:07.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1780aaf60 con 0x7fe1780a9ac0 2024-03-29T11:41:07.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:07.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe17d0ce640 10 monclient(hunting): authenticate will time out at 1257.177417s 2024-03-29T11:41:07.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe176d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe1780b4100 0x7fe17813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:41776/0 (socket says 172.21.15.145:41776) 2024-03-29T11:41:07.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.869+0000 7fe176d9d640 1 -- 172.21.15.145:0/192876149 learned_addr learned my addr 172.21.15.145:0/192876149 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:07.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724888005 0 0) 0x7fe1780aac80 con 0x7fe1780b4100 2024-03-29T11:41:07.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:07.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:07.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:07.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:41:07.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe160005730 con 0x7fe1780b4100 2024-03-29T11:41:07.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488201882 0 0) 0x7fe1780aaf60 con 0x7fe1780a9ac0 2024-03-29T11:41:07.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:41:07.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:07.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:07.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:41:07.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1780aac80 con 0x7fe1780a9ac0 2024-03-29T11:41:07.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.873+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2158896130 0 0) 0x7fe1780aac80 con 0x7fe1780a9ac0 2024-03-29T11:41:07.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 >> v1:172.21.15.203:6789/0 conn(0x7fe1780b4100 legacy=0x7fe17813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:07.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 >> v1:172.21.15.145:6789/0 conn(0x7fe17813ee90 legacy=0x7fe178146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:07.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:41:07.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 1 monclient: found mon.noname-b 2024-03-29T11:41:07.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:07.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:41:07.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17813e970 con 0x7fe1780a9ac0 2024-03-29T11:41:07.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 10 monclient: _finish_auth 0 2024-03-29T11:41:07.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:41:07.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:07.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe17d0ce640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:41:07.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe17d0ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:07.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe1700019f0 con 0x7fe1780a9ac0 2024-03-29T11:41:07.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 10 monclient: dump: 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:07.899 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:07.900 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:07.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe17d0ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:07.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe170001c10 con 0x7fe1780a9ac0 2024-03-29T11:41:07.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe175d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:07.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe17d0ce640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:07.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.874+0000 7fe17d0ce640 4 set_mon_vals no callback set 2024-03-29T11:41:07.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.875+0000 7fe175d9b640 1 -- 172.21.15.145:0/192876149 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe170001fa0 con 0x7fe1780a9ac0 2024-03-29T11:41:07.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.875+0000 7fe175d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:07.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.875+0000 7fe175d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:07.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.875+0000 7fe175d9b640 10 monclient: dump: 2024-03-29T11:41:07.901 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:07.901 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:07.901 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.901 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.901 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:07.901 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:07.901 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:07.901 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.875+0000 7fe17d0ce640 10 monclient: shutdown 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.875+0000 7fe17d0ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.875+0000 7fe17d0ce640 1 -- 172.21.15.145:0/192876149 >> v1:172.21.15.145:6790/0 conn(0x7fe1780a9ac0 legacy=0x7fe17814a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.875+0000 7fe17d0ce640 1 -- 172.21.15.145:0/192876149 shutdown_connections 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.875+0000 7fe17d0ce640 1 -- 172.21.15.145:0/192876149 >> 172.21.15.145:0/192876149 conn(0x7fe1780b35e0 msgr2=0x7fe1780b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.876+0000 7fe17d0ce640 1 -- 172.21.15.145:0/192876149 shutdown_connections 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.876+0000 7fe17d0ce640 1 -- 172.21.15.145:0/192876149 wait complete. 2024-03-29T11:41:07.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.877+0000 7fe17d0ce640 10 monclient: build_initial_monmap 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.877+0000 7fe17d0ce640 10 monclient: monmap: 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:41:07.903 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:41:07.904 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:07.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.877+0000 7fe17d0ce640 1 librados: starting msgr at 2024-03-29T11:41:07.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.877+0000 7fe17d0ce640 1 librados: starting objecter 2024-03-29T11:41:07.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.878+0000 7fe17d0ce640 1 Processor -- start 2024-03-29T11:41:07.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.878+0000 7fe17d0ce640 1 -- start start 2024-03-29T11:41:07.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.878+0000 7fe17d0ce640 1 librados: setting wanted keys 2024-03-29T11:41:07.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.878+0000 7fe17d0ce640 1 librados: calling monclient init 2024-03-29T11:41:07.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.878+0000 7fe17d0ce640 10 monclient: init 2024-03-29T11:41:07.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.879+0000 7fe17d0ce640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:07.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.879+0000 7fe17d0ce640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:07.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.879+0000 7fe17d0ce640 10 monclient(hunting): picked mon.noname-a con 0x7fe1780a9ac0 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:07.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.879+0000 7fe17d0ce640 10 monclient(hunting): picked mon.noname-c con 0x7fe1780b4100 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:07.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe17d0ce640 10 monclient(hunting): picked mon.noname-b con 0x7fe17813ee90 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:07.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe17d0ce640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1781d8370 con 0x7fe1780a9ac0 2024-03-29T11:41:07.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe17d0ce640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1781d7dc0 con 0x7fe17813ee90 2024-03-29T11:41:07.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe17d0ce640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1781d7f50 con 0x7fe1780b4100 2024-03-29T11:41:07.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe17d0ce640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:07.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe17d0ce640 10 monclient(hunting): authenticate will time out at 1257.189509s 2024-03-29T11:41:07.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe17759e640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe17813ee90 0x7fe1781dacf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:42042/0 (socket says 172.21.15.145:42042) 2024-03-29T11:41:07.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe17759e640 1 -- 172.21.15.145:0/4146480895 learned_addr learned my addr 172.21.15.145:0/4146480895 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:07.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe15f7fe640 10 client.?.objecter ms_handle_connect 0x7fe17813ee90 2024-03-29T11:41:07.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.881+0000 7fe15f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:07.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4029479662 0 0) 0x7fe1781d7dc0 con 0x7fe17813ee90 2024-03-29T11:41:07.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:07.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:07.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:07.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:41:07.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe164004eb0 con 0x7fe17813ee90 2024-03-29T11:41:07.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209191358 0 0) 0x7fe164004eb0 con 0x7fe17813ee90 2024-03-29T11:41:07.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1781d7dc0 con 0x7fe17813ee90 2024-03-29T11:41:07.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe1700019f0 con 0x7fe17813ee90 2024-03-29T11:41:07.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:07.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:07.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 10 monclient(hunting): dump: 2024-03-29T11:41:07.910 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:07.910 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:07.910 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.910 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.910 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:07.910 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:07.910 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:07.910 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:07.911 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:07.911 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:07.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.882+0000 7fe15f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:07.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.884+0000 7fe15f7fe640 10 client.?.objecter ms_handle_connect 0x7fe1780b4100 2024-03-29T11:41:07.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.884+0000 7fe15f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:07.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.884+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285326302 0 0) 0x7fe1781d7f50 con 0x7fe1780b4100 2024-03-29T11:41:07.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.884+0000 7fe15f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:07.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.884+0000 7fe15f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:07.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.884+0000 7fe15f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.884+0000 7fe15f7fe640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.884+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe164004eb0 con 0x7fe1780b4100 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.884+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3376591349 0 0) 0x7fe164004eb0 con 0x7fe1780b4100 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.885+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe164005990 con 0x7fe1780b4100 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.885+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe168001710 con 0x7fe1780b4100 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.885+0000 7fe15f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.885+0000 7fe15f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.885+0000 7fe15f7fe640 10 monclient(hunting): dump: 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.912 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.885+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 68097555 0 0) 0x7fe1781d7dc0 con 0x7fe17813ee90 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 >> v1:172.21.15.145:6790/0 conn(0x7fe1780b4100 legacy=0x7fe178139c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 >> v1:172.21.15.145:6789/0 conn(0x7fe1780a9ac0 legacy=0x7fe1781d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 1 monclient: found mon.b 2024-03-29T11:41:07.913 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:07.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:07.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1781d80e0 con 0x7fe17813ee90 2024-03-29T11:41:07.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:07.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:07.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe15f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:07.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe17d0ce640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:41:07.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe17d0ce640 10 monclient: _renew_subs 2024-03-29T11:41:07.914 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe17d0ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:07.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe17813a340 con 0x7fe17813ee90 2024-03-29T11:41:07.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe17d0ce640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:07.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe17d0ce640 10 monclient: _renew_subs 2024-03-29T11:41:07.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe17d0ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:07.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe178142280 con 0x7fe17813ee90 2024-03-29T11:41:07.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.886+0000 7fe17d0ce640 1 librados: init done 2024-03-29T11:41:07.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.888+0000 7fe17d0ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:07.916 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.888+0000 7fe17d0ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:07.916 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.888+0000 7fe17d0ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:07.916 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.888+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe178141fb0 con 0x7fe17813ee90 2024-03-29T11:41:07.916 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.889+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe170003c20 con 0x7fe17813ee90 2024-03-29T11:41:07.916 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.889+0000 7fe15f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:07.916 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.889+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe170004030 con 0x7fe17813ee90 2024-03-29T11:41:07.916 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.889+0000 7fe15f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:07.916 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.889+0000 7fe15f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:07.917 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.889+0000 7fe15f7fe640 10 monclient: dump: 2024-03-29T11:41:07.917 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:07.917 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:07.917 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.917 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:07.917 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:07.917 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:07.918 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:07.918 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:07.918 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:07.918 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:07.918 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.889+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fe170004250 con 0x7fe17813ee90 2024-03-29T11:41:07.918 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.889+0000 7fe17559a640 4 set_mon_vals no callback set 2024-03-29T11:41:07.918 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.890+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fe1700850e0 con 0x7fe17813ee90 2024-03-29T11:41:07.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.890+0000 7fe15f7fe640 10 client.4444.objecter ms_dispatch 0x7fe178000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:07.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.890+0000 7fe15f7fe640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:07.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.890+0000 7fe15f7fe640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:07.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.890+0000 7fe15f7fe640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-29T11:41:07.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.891+0000 7fe176d9d640 10 monclient: get_auth_request con 0x7fe16406e2e0 auth_method 0 2024-03-29T11:41:07.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.891+0000 7fe15f7fe640 10 client.4444.objecter ms_handle_connect 0x7fe16406e2e0 2024-03-29T11:41:07.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.898+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe170050a40 con 0x7fe17813ee90 2024-03-29T11:41:07.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.898+0000 7fe15f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:07.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:07.898+0000 7fe15f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:08.225 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.223+0000 7fe17d0ce640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:08.226 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.223+0000 7fe17d0ce640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:08.226 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.223+0000 7fe17d0ce640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:08.226 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.223+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1781daa10 con 0x7fe17813ee90 2024-03-29T11:41:08.226 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.224+0000 7fe15f7fe640 1 -- 172.21.15.145:0/4146480895 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fe170052970 con 0x7fe17813ee90 2024-03-29T11:41:08.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.224+0000 7fe15f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:08.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.224+0000 7fe15f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:08.227 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:41:08.227 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:08.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.227+0000 7fe17d0ce640 10 librados: watch_flush enter 2024-03-29T11:41:08.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.227+0000 7fe17d0ce640 10 librados: watch_flush exit 2024-03-29T11:41:08.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.227+0000 7fe17d0ce640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-29T11:41:08.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.227+0000 7fe17d0ce640 10 client.4444.objecter successfully canceled tick 2024-03-29T11:41:08.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.228+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 >> v1:172.21.15.145:6816/227372239 conn(0x7fe16406e2e0 legacy=0x7fe1640707a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:08.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.228+0000 7fe17d0ce640 10 monclient: shutdown 2024-03-29T11:41:08.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.228+0000 7fe17d0ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:08.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.228+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 >> v1:172.21.15.203:6789/0 conn(0x7fe17813ee90 legacy=0x7fe1781dacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:08.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.228+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 shutdown_connections 2024-03-29T11:41:08.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.228+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 >> 172.21.15.145:0/4146480895 conn(0x7fe1780b35e0 msgr2=0x7fe1780ad0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:08.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.228+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 shutdown_connections 2024-03-29T11:41:08.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.228+0000 7fe17d0ce640 1 -- 172.21.15.145:0/4146480895 wait complete. 2024-03-29T11:41:08.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:08.228+0000 7fe17d0ce640 1 librados: shutdown 2024-03-29T11:41:08.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:15.057 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:15.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.213+0000 7f88b52e2640 10 monclient: get_monmap_and_config 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.213+0000 7f88b52e2640 10 monclient: build_initial_monmap 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.213+0000 7f88b52e2640 10 monclient: monmap: 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:41:15.214590+0000 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:41:15.214590+0000 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:41:15.216 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:15.220 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.216+0000 7f88b52e2640 1 Processor -- start 2024-03-29T11:41:15.220 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.216+0000 7f88b52e2640 1 -- start start 2024-03-29T11:41:15.220 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.216+0000 7f88b52e2640 10 monclient: init 2024-03-29T11:41:15.220 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:15.220 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:15.220 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 10 monclient(hunting): picked mon.noname-b con 0x7f88b00b1f60 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:15.220 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 10 monclient(hunting): picked mon.noname-c con 0x7f88b0071eb0 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:15.221 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 10 monclient(hunting): picked mon.noname-a con 0x7f88b0143d10 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:15.221 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b006ef80 con 0x7f88b0143d10 2024-03-29T11:41:15.221 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b006f260 con 0x7f88b0071eb0 2024-03-29T11:41:15.221 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00b40c0 con 0x7f88b00b1f60 2024-03-29T11:41:15.221 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:15.221 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.217+0000 7f88b52e2640 10 monclient(hunting): authenticate will time out at 1264.525878s 2024-03-29T11:41:15.225 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.223+0000 7f88ae7fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f88b0071eb0 0x7f88b0146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:43767/0 (socket says 172.21.15.145:43767) 2024-03-29T11:41:15.225 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.223+0000 7f88ae7fc640 1 -- 172.21.15.145:0/3272430049 learned_addr learned my addr 172.21.15.145:0/3272430049 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305064747 0 0) 0x7f88b00b40c0 con 0x7f88b00b1f60 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8898003520 con 0x7f88b00b1f60 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870536698 0 0) 0x7f88b006f260 con 0x7f88b0071eb0 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient(hunting): _negotiate 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:15.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:15.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:41:15.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b00b40c0 con 0x7f88b0071eb0 2024-03-29T11:41:15.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 33658862 0 0) 0x7f8898003520 con 0x7f88b00b1f60 2024-03-29T11:41:15.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 >> v1:172.21.15.203:6789/0 conn(0x7f88b0071eb0 legacy=0x7f88b0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:15.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 >> v1:172.21.15.145:6789/0 conn(0x7f88b0143d10 legacy=0x7f88b0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:15.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient: _finish_hunting 0 2024-03-29T11:41:15.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 monclient: found mon.noname-b 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b0155d70 con 0x7f88b00b1f60 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient: _finish_auth 0 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient: _check_auth_tickets 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f88a8001a10 con 0x7f88b00b1f60 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.225+0000 7f88adffb640 10 monclient: dump: 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:15.231 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.226+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88a8001b70 con 0x7f88b00b1f60 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.226+0000 7f88adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.226+0000 7f88adffb640 1 -- 172.21.15.145:0/3272430049 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f88a8001f80 con 0x7f88b00b1f60 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.226+0000 7f88adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.226+0000 7f88adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.226+0000 7f88adffb640 10 monclient: dump: 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:15.232 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:15.233 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:15.233 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:15.233 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:15.233 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:15.233 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:15.233 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:15.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.227+0000 7f88b52e2640 5 monclient: authenticate success, global_id 4469 2024-03-29T11:41:15.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.227+0000 7f88b52e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:15.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.227+0000 7f88b52e2640 4 set_mon_vals no callback set 2024-03-29T11:41:15.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.228+0000 7f88b52e2640 10 monclient: shutdown 2024-03-29T11:41:15.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.228+0000 7f88b52e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:15.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.228+0000 7f88b52e2640 1 -- 172.21.15.145:0/3272430049 >> v1:172.21.15.145:6790/0 conn(0x7f88b00b1f60 legacy=0x7f88b0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:15.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.231+0000 7f88b52e2640 1 -- 172.21.15.145:0/3272430049 shutdown_connections 2024-03-29T11:41:15.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.231+0000 7f88b52e2640 1 -- 172.21.15.145:0/3272430049 >> 172.21.15.145:0/3272430049 conn(0x7f88b0066a00 msgr2=0x7f88b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:15.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.231+0000 7f88b52e2640 1 -- 172.21.15.145:0/3272430049 shutdown_connections 2024-03-29T11:41:15.237 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.231+0000 7f88b52e2640 1 -- 172.21.15.145:0/3272430049 wait complete. 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.232+0000 7f88b52e2640 10 monclient: build_initial_monmap 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.232+0000 7f88b52e2640 10 monclient: monmap: 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.233+0000 7f88b52e2640 1 librados: starting msgr at 2024-03-29T11:41:15.238 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.233+0000 7f88b52e2640 1 librados: starting objecter 2024-03-29T11:41:15.239 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.233+0000 7f88b52e2640 1 Processor -- start 2024-03-29T11:41:15.244 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.237+0000 7f88b52e2640 1 -- start start 2024-03-29T11:41:15.244 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.237+0000 7f88b52e2640 1 librados: setting wanted keys 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.237+0000 7f88b52e2640 1 librados: calling monclient init 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.237+0000 7f88b52e2640 10 monclient: init 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 10 monclient(hunting): picked mon.noname-c con 0x7f88b00b1f60 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 10 monclient(hunting): picked mon.noname-b con 0x7f88b0143d10 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 10 monclient(hunting): picked mon.noname-a con 0x7f88b0070900 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00ab650 con 0x7f88b0070900 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00ac8e0 con 0x7f88b0143d10 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00acb50 con 0x7f88b00b1f60 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.238+0000 7f88b52e2640 10 monclient(hunting): authenticate will time out at 1264.546498s 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88ae7fc640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f88b0143d10 0x7f88b013fce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:43849/0 (socket says 172.21.15.145:43849) 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88ae7fc640 1 -- 172.21.15.145:0/432231634 learned_addr learned my addr 172.21.15.145:0/432231634 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:15.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88977fe640 10 client.?.objecter ms_handle_connect 0x7f88b0143d10 2024-03-29T11:41:15.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:15.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452549868 0 0) 0x7f88b00ac8e0 con 0x7f88b0143d10 2024-03-29T11:41:15.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:15.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:15.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:15.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88977fe640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:41:15.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.242+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f889c004eb0 con 0x7f88b0143d10 2024-03-29T11:41:15.247 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 client.?.objecter ms_handle_connect 0x7f88b0070900 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4113967095 0 0) 0x7f889c004eb0 con 0x7f88b0143d10 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88b00ac8e0 con 0x7f88b0143d10 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f88a40019b0 con 0x7f88b0143d10 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 monclient(hunting): dump: 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:15.248 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148116935 0 0) 0x7f88b00ab650 con 0x7f88b0070900 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f889c004eb0 con 0x7f88b0070900 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2709667091 0 0) 0x7f88b00ac8e0 con 0x7f88b0143d10 2024-03-29T11:41:15.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.243+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 >> v1:172.21.15.145:6790/0 conn(0x7f88b00b1f60 legacy=0x7f88b00aad60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:15.250 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.246+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 >> v1:172.21.15.145:6789/0 conn(0x7f88b0070900 legacy=0x7f88b01e3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:15.250 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.246+0000 7f88977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.246+0000 7f88977fe640 1 monclient: found mon.b 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.246+0000 7f88977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88977fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b01403f0 con 0x7f88b0143d10 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88977fe640 10 monclient: _finish_auth 0 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88b52e2640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88b52e2640 10 monclient: _renew_subs 2024-03-29T11:41:15.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88b52e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:15.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b01405e0 con 0x7f88b0143d10 2024-03-29T11:41:15.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88b52e2640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:15.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88b52e2640 10 monclient: _renew_subs 2024-03-29T11:41:15.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88b52e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:15.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b01e4870 con 0x7f88b0143d10 2024-03-29T11:41:15.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.247+0000 7f88b52e2640 1 librados: init done 2024-03-29T11:41:15.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.249+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88a40047e0 con 0x7f88b0143d10 2024-03-29T11:41:15.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.250+0000 7f88977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.250+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f88a4004bb0 con 0x7f88b0143d10 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.250+0000 7f88977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.250+0000 7f88977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.251+0000 7f88acff9640 4 set_mon_vals no callback set 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.251+0000 7f88977fe640 10 monclient: dump: 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:15.254 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:15.255 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:15.255 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:15.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.251+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f88a4004dd0 con 0x7f88b0143d10 2024-03-29T11:41:15.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f88a4084bd0 con 0x7f88b0143d10 2024-03-29T11:41:15.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88977fe640 10 client.4450.objecter ms_dispatch 0x7f88b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:15.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88b52e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:15.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88977fe640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:15.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88977fe640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:15.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88977fe640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-29T11:41:15.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88b52e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:15.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88b52e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:15.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890005080 con 0x7f88b0143d10 2024-03-29T11:41:15.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.252+0000 7f88aeffd640 10 monclient: get_auth_request con 0x7f889c06df30 auth_method 0 2024-03-29T11:41:15.259 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.256+0000 7f88977fe640 10 client.4450.objecter ms_handle_connect 0x7f889c06df30 2024-03-29T11:41:15.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.263+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88a4050530 con 0x7f88b0143d10 2024-03-29T11:41:15.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.263+0000 7f88977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:15.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.263+0000 7f88977fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:15.590 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.587+0000 7f88b52e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:15.590 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.587+0000 7f88b52e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:15.590 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.587+0000 7f88b52e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:15.590 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.587+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8890005330 con 0x7f88b0143d10 2024-03-29T11:41:15.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.589+0000 7f88977fe640 1 -- 172.21.15.145:0/432231634 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f88a4052460 con 0x7f88b0143d10 2024-03-29T11:41:15.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.589+0000 7f88977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:15.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.589+0000 7f88977fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:15.592 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:41:15.593 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:15.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.592+0000 7f88b52e2640 10 librados: watch_flush enter 2024-03-29T11:41:15.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.593+0000 7f88b52e2640 10 librados: watch_flush exit 2024-03-29T11:41:15.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.593+0000 7f88b52e2640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-29T11:41:15.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.593+0000 7f88b52e2640 10 client.4450.objecter successfully canceled tick 2024-03-29T11:41:15.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.595+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 >> v1:172.21.15.145:6816/227372239 conn(0x7f889c06df30 legacy=0x7f889c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:15.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.595+0000 7f88b52e2640 10 monclient: shutdown 2024-03-29T11:41:15.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.595+0000 7f88b52e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:15.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.595+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 >> v1:172.21.15.203:6789/0 conn(0x7f88b0143d10 legacy=0x7f88b013fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:15.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.596+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 shutdown_connections 2024-03-29T11:41:15.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.596+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 >> 172.21.15.145:0/432231634 conn(0x7f88b0066a00 msgr2=0x7f88b00b5c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:15.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.596+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 shutdown_connections 2024-03-29T11:41:15.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.596+0000 7f88b52e2640 1 -- 172.21.15.145:0/432231634 wait complete. 2024-03-29T11:41:15.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:15.596+0000 7f88b52e2640 1 librados: shutdown 2024-03-29T11:41:15.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:22.430 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:22.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.593+0000 7f600a095640 10 monclient: get_monmap_and_config 2024-03-29T11:41:22.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.593+0000 7f600a095640 10 monclient: build_initial_monmap 2024-03-29T11:41:22.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.593+0000 7f600a095640 10 monclient: monmap: 2024-03-29T11:41:22.596 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:22.597 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:22.597 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:41:22.594830+0000 2024-03-29T11:41:22.597 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:41:22.594830+0000 2024-03-29T11:41:22.597 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:22.597 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:22.597 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:22.597 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:41:22.597 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:41:22.597 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:22.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.595+0000 7f600a095640 1 Processor -- start 2024-03-29T11:41:22.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.596+0000 7f600a095640 1 -- start start 2024-03-29T11:41:22.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.596+0000 7f600a095640 10 monclient: init 2024-03-29T11:41:22.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 10 monclient(hunting): picked mon.noname-c con 0x7f60040b4710 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 10 monclient(hunting): picked mon.noname-b con 0x7f60040b3970 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 10 monclient(hunting): picked mon.noname-a con 0x7f600413f390 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60040af020 con 0x7f600413f390 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60040b0910 con 0x7f60040b4710 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60040b0bf0 con 0x7f60040b3970 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f600a095640 10 monclient(hunting): authenticate will time out at 1271.905382s 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f6003fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f600413f390 0x7f600413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:52996/0 (socket says 172.21.15.145:52996) 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.597+0000 7f6003fff640 1 -- 172.21.15.145:0/536786623 learned_addr learned my addr 172.21.15.145:0/536786623 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 1 -- 172.21.15.145:0/536786623 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966248425 0 0) 0x7f60040af020 con 0x7f600413f390 2024-03-29T11:41:22.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 1 -- 172.21.15.145:0/536786623 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec003520 con 0x7f600413f390 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 1 -- 172.21.15.145:0/536786623 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3588578644 0 0) 0x7f5fec003520 con 0x7f600413f390 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 1 -- 172.21.15.145:0/536786623 >> v1:172.21.15.145:6790/0 conn(0x7f60040b3970 legacy=0x7f60040af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 1 -- 172.21.15.145:0/536786623 >> v1:172.21.15.203:6789/0 conn(0x7f60040b4710 legacy=0x7f60040b1010 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 1 monclient: found mon.noname-a 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 1 -- 172.21.15.145:0/536786623 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60040aeb00 con 0x7f600413f390 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:22.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 1 -- 172.21.15.145:0/536786623 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ff8001690 con 0x7f600413f390 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f60027fc640 10 monclient: dump: 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:22.607 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f600a095640 5 monclient: authenticate success, global_id 4449 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.598+0000 7f600a095640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.599+0000 7f60027fc640 1 -- 172.21.15.145:0/536786623 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff8001860 con 0x7f600413f390 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.599+0000 7f60027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.599+0000 7f60027fc640 1 -- 172.21.15.145:0/536786623 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ff8001c70 con 0x7f600413f390 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.599+0000 7f60027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.599+0000 7f60027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.599+0000 7f60027fc640 10 monclient: dump: 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:22.608 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.599+0000 7f600a095640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.599+0000 7f600a095640 4 set_mon_vals no callback set 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.600+0000 7f600a095640 10 monclient: shutdown 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.600+0000 7f600a095640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.600+0000 7f600a095640 1 -- 172.21.15.145:0/536786623 >> v1:172.21.15.145:6789/0 conn(0x7f600413f390 legacy=0x7f600413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.600+0000 7f600a095640 1 -- 172.21.15.145:0/536786623 shutdown_connections 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.600+0000 7f600a095640 1 -- 172.21.15.145:0/536786623 >> 172.21.15.145:0/536786623 conn(0x7f6004066a00 msgr2=0x7f6004066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.600+0000 7f600a095640 1 -- 172.21.15.145:0/536786623 shutdown_connections 2024-03-29T11:41:22.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.600+0000 7f600a095640 1 -- 172.21.15.145:0/536786623 wait complete. 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.602+0000 7f600a095640 10 monclient: build_initial_monmap 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.603+0000 7f600a095640 10 monclient: monmap: 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:41:22.616 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.603+0000 7f600a095640 1 librados: starting msgr at 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.603+0000 7f600a095640 1 librados: starting objecter 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.604+0000 7f600a095640 1 Processor -- start 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.604+0000 7f600a095640 1 -- start start 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.604+0000 7f600a095640 1 librados: setting wanted keys 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.604+0000 7f600a095640 1 librados: calling monclient init 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.604+0000 7f600a095640 10 monclient: init 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 10 monclient(hunting): picked mon.noname-b con 0x7f60040b3970 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 10 monclient(hunting): picked mon.noname-c con 0x7f60040b4710 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 10 monclient(hunting): picked mon.noname-a con 0x7f600413f390 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f600406f840 con 0x7f600413f390 2024-03-29T11:41:22.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60041db6d0 con 0x7f60040b3970 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60041357f0 con 0x7f60040b4710 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f600a095640 10 monclient(hunting): authenticate will time out at 1271.913406s 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f6002ffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f60040b4710 0x7f60041dbbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:34912/0 (socket says 172.21.15.145:34912) 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f6002ffd640 1 -- 172.21.15.145:0/3529226866 learned_addr learned my addr 172.21.15.145:0/3529226866 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f5febfff640 10 client.?.objecter ms_handle_connect 0x7f60040b4710 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f5febfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f5febfff640 10 client.?.objecter ms_handle_connect 0x7f60040b3970 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.605+0000 7f5febfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.606+0000 7f5febfff640 10 client.?.objecter ms_handle_connect 0x7f600413f390 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.606+0000 7f5febfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:22.618 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282722138 0 0) 0x7f60041db6d0 con 0x7f60040b3970 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff0004ee0 con 0x7f60040b3970 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085566334 0 0) 0x7f60041357f0 con 0x7f60040b4710 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60041db6d0 con 0x7f60040b4710 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745500288 0 0) 0x7f600406f840 con 0x7f600413f390 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:22.619 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60041357f0 con 0x7f600413f390 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3999239623 0 0) 0x7f5ff0004ee0 con 0x7f60040b3970 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f600406f840 con 0x7f60040b3970 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ffc001b20 con 0x7f60040b3970 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.607+0000 7f5febfff640 10 monclient(hunting): dump: 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:22.620 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1198851909 0 0) 0x7f60041357f0 con 0x7f600413f390 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ff0004ee0 con 0x7f600413f390 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3882961783 0 0) 0x7f600406f840 con 0x7f60040b3970 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 >> v1:172.21.15.145:6790/0 conn(0x7f60040b4710 legacy=0x7f60041dbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 >> v1:172.21.15.145:6789/0 conn(0x7f600413f390 legacy=0x7f60041e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:22.621 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 1 monclient: found mon.b 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.608+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f600406e810 con 0x7f60040b3970 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f5febfff640 10 monclient: _finish_auth 0 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f5febfff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f5febfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f600a095640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f600a095640 10 monclient: _renew_subs 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f600a095640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6004138180 con 0x7f60040b3970 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ffc002030 con 0x7f60040b3970 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f5febfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ffc003bb0 con 0x7f60040b3970 2024-03-29T11:41:22.622 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f5febfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f5febfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f5febfff640 10 monclient: dump: 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.609+0000 7f60017fa640 4 set_mon_vals no callback set 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.610+0000 7f600a095640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.610+0000 7f600a095640 10 monclient: _renew_subs 2024-03-29T11:41:22.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.610+0000 7f600a095640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.610+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60041dc4e0 con 0x7f60040b3970 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.610+0000 7f600a095640 1 librados: init done 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.612+0000 7f600a095640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.612+0000 7f600a095640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.612+0000 7f600a095640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.612+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60040688a0 con 0x7f60040b3970 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.612+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f5ffc019b80 con 0x7f60040b3970 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.613+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f5ffc083a60 con 0x7f60040b3970 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.613+0000 7f5febfff640 10 client.4453.objecter ms_dispatch 0x7f6004000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.613+0000 7f5febfff640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.613+0000 7f5febfff640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.613+0000 7f5febfff640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.613+0000 7f6002ffd640 10 monclient: get_auth_request con 0x7f5ff006dec0 auth_method 0 2024-03-29T11:41:22.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.613+0000 7f5febfff640 10 client.4453.objecter ms_handle_connect 0x7f5ff006dec0 2024-03-29T11:41:22.625 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.619+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ffc04f340 con 0x7f60040b3970 2024-03-29T11:41:22.625 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.619+0000 7f5febfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:22.625 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:22.619+0000 7f5febfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:23.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.018+0000 7f600a095640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:23.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.018+0000 7f600a095640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:23.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.018+0000 7f600a095640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:23.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.018+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60040b54d0 con 0x7f60040b3970 2024-03-29T11:41:23.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.019+0000 7f5febfff640 1 -- 172.21.15.145:0/3529226866 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f5ffc051270 con 0x7f60040b3970 2024-03-29T11:41:23.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.019+0000 7f5febfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:23.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.019+0000 7f5febfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:23.023 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:41:23.023 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:23.025 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 10 librados: watch_flush enter 2024-03-29T11:41:23.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 10 librados: watch_flush exit 2024-03-29T11:41:23.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:41:23.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:41:23.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 >> v1:172.21.15.145:6816/227372239 conn(0x7f5ff006dec0 legacy=0x7f5ff0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:23.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 10 monclient: shutdown 2024-03-29T11:41:23.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:23.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 >> v1:172.21.15.203:6789/0 conn(0x7f60040b3970 legacy=0x7f600406e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:23.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 shutdown_connections 2024-03-29T11:41:23.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 >> 172.21.15.145:0/3529226866 conn(0x7f6004066a00 msgr2=0x7f60040b7290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:23.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 shutdown_connections 2024-03-29T11:41:23.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.023+0000 7f600a095640 1 -- 172.21.15.145:0/3529226866 wait complete. 2024-03-29T11:41:23.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:23.024+0000 7f600a095640 1 librados: shutdown 2024-03-29T11:41:23.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:29.849 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:30.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.000+0000 7faa586c0640 10 monclient: get_monmap_and_config 2024-03-29T11:41:30.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.000+0000 7faa586c0640 10 monclient: build_initial_monmap 2024-03-29T11:41:30.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.000+0000 7faa586c0640 10 monclient: monmap: 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:41:30.001099+0000 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:41:30.001099+0000 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:41:30.003 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:30.006 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.003+0000 7faa586c0640 1 Processor -- start 2024-03-29T11:41:30.006 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.003+0000 7faa586c0640 1 -- start start 2024-03-29T11:41:30.006 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.003+0000 7faa586c0640 10 monclient: init 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 10 monclient(hunting): picked mon.noname-a con 0x7faa48006e50 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 10 monclient(hunting): picked mon.noname-c con 0x7faa48035e20 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 10 monclient(hunting): picked mon.noname-b con 0x7faa480c9690 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa48035990 con 0x7faa48006e50 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480362a0 con 0x7faa48035e20 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa48035660 con 0x7faa480c9690 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:30.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.004+0000 7faa586c0640 10 monclient(hunting): authenticate will time out at 1279.311558s 2024-03-29T11:41:30.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.008+0000 7faa56c5d640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7faa480c9690 0x7faa480d7580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:50927/0 (socket says 172.21.15.145:50927) 2024-03-29T11:41:30.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.008+0000 7faa56c5d640 1 -- 172.21.15.145:0/3579284092 learned_addr learned my addr 172.21.15.145:0/3579284092 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507252385 0 0) 0x7faa48035660 con 0x7faa480c9690 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa3c003520 con 0x7faa480c9690 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620685538 0 0) 0x7faa48035990 con 0x7faa48006e50 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:30.013 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:30.014 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:41:30.014 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.011+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa48035660 con 0x7faa48006e50 2024-03-29T11:41:30.014 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428305652 0 0) 0x7faa480362a0 con 0x7faa48035e20 2024-03-29T11:41:30.014 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:30.014 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa48035990 con 0x7faa48035e20 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 803514524 0 0) 0x7faa48035660 con 0x7faa48006e50 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 >> v1:172.21.15.145:6790/0 conn(0x7faa480c9690 legacy=0x7faa480d7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 >> v1:172.21.15.203:6789/0 conn(0x7faa48035e20 legacy=0x7faa480c8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient: _finish_hunting 0 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 1 monclient: found mon.noname-a 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa48034e40 con 0x7faa48006e50 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient: _finish_auth 0 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient: _check_auth_tickets 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:30.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faa4c001690 con 0x7faa48006e50 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa5545a640 10 monclient: dump: 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:30.016 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa586c0640 5 monclient: authenticate success, global_id 4455 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.012+0000 7faa586c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.013+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa4c0017f0 con 0x7faa48006e50 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.013+0000 7faa5545a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.013+0000 7faa586c0640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.013+0000 7faa586c0640 4 set_mon_vals no callback set 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.013+0000 7faa5545a640 1 -- 172.21.15.145:0/3579284092 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faa4c001c00 con 0x7faa48006e50 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.013+0000 7faa5545a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.013+0000 7faa5545a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:30.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.013+0000 7faa5545a640 10 monclient: dump: 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:30.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.016+0000 7faa586c0640 10 monclient: shutdown 2024-03-29T11:41:30.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.016+0000 7faa586c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:30.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.016+0000 7faa586c0640 1 -- 172.21.15.145:0/3579284092 >> v1:172.21.15.145:6789/0 conn(0x7faa48006e50 legacy=0x7faa480cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:30.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.018+0000 7faa586c0640 1 -- 172.21.15.145:0/3579284092 shutdown_connections 2024-03-29T11:41:30.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.018+0000 7faa586c0640 1 -- 172.21.15.145:0/3579284092 >> 172.21.15.145:0/3579284092 conn(0x7faa48005c20 msgr2=0x7faa48006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:30.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.019+0000 7faa586c0640 1 -- 172.21.15.145:0/3579284092 shutdown_connections 2024-03-29T11:41:30.023 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.020+0000 7faa586c0640 1 -- 172.21.15.145:0/3579284092 wait complete. 2024-03-29T11:41:30.023 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.020+0000 7faa586c0640 10 monclient: build_initial_monmap 2024-03-29T11:41:30.023 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.020+0000 7faa586c0640 10 monclient: monmap: 2024-03-29T11:41:30.023 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:30.023 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:30.023 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.021+0000 7faa586c0640 1 librados: starting msgr at 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.021+0000 7faa586c0640 1 librados: starting objecter 2024-03-29T11:41:30.024 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.021+0000 7faa586c0640 1 Processor -- start 2024-03-29T11:41:30.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.021+0000 7faa586c0640 1 -- start start 2024-03-29T11:41:30.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.021+0000 7faa586c0640 1 librados: setting wanted keys 2024-03-29T11:41:30.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.021+0000 7faa586c0640 1 librados: calling monclient init 2024-03-29T11:41:30.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.021+0000 7faa586c0640 10 monclient: init 2024-03-29T11:41:30.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:30.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 10 monclient(hunting): picked mon.noname-b con 0x7faa48006e50 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 10 monclient(hunting): picked mon.noname-a con 0x7faa48035e20 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 10 monclient(hunting): picked mon.noname-c con 0x7faa480c9690 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480cd7f0 con 0x7faa48035e20 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480cdad0 con 0x7faa48006e50 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa480cddb0 con 0x7faa480c9690 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.022+0000 7faa586c0640 10 monclient(hunting): authenticate will time out at 1279.329826s 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.026+0000 7faa55c5b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7faa48035e20 0x7faa480c0b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:57445/0 (socket says 172.21.15.145:57445) 2024-03-29T11:41:30.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.026+0000 7faa55c5b640 1 -- 172.21.15.145:0/3308833504 learned_addr learned my addr 172.21.15.145:0/3308833504 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:30.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.027+0000 7faa46ffd640 10 client.?.objecter ms_handle_connect 0x7faa48035e20 2024-03-29T11:41:30.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.027+0000 7faa46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:30.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.027+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573114575 0 0) 0x7faa480cd7f0 con 0x7faa48035e20 2024-03-29T11:41:30.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.028+0000 7faa46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:30.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.028+0000 7faa46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:30.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.028+0000 7faa46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:30.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.028+0000 7faa46ffd640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:41:30.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.028+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa34004eb0 con 0x7faa48035e20 2024-03-29T11:41:30.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.030+0000 7faa46ffd640 10 client.?.objecter ms_handle_connect 0x7faa480c9690 2024-03-29T11:41:30.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.030+0000 7faa46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:30.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.030+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131385169 0 0) 0x7faa480cddb0 con 0x7faa480c9690 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.030+0000 7faa46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.030+0000 7faa46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.030+0000 7faa46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.030+0000 7faa46ffd640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.030+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa480cd7f0 con 0x7faa480c9690 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.031+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4124997806 0 0) 0x7faa480cd7f0 con 0x7faa480c9690 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.031+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa480cddb0 con 0x7faa480c9690 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.031+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faa5004fdd0 con 0x7faa480c9690 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.031+0000 7faa46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.031+0000 7faa46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.031+0000 7faa46ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:30.038 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.031+0000 7faa46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 client.?.objecter ms_handle_connect 0x7faa48006e50 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524604996 0 0) 0x7faa480cdad0 con 0x7faa48006e50 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:30.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa480cd7f0 con 0x7faa48006e50 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3317804028 0 0) 0x7faa34004eb0 con 0x7faa48035e20 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa480cdad0 con 0x7faa48035e20 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3122366199 0 0) 0x7faa480cddb0 con 0x7faa480c9690 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 >> v1:172.21.15.203:6789/0 conn(0x7faa48006e50 legacy=0x7faa48033810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 >> v1:172.21.15.145:6789/0 conn(0x7faa48035e20 legacy=0x7faa480c0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 1 monclient: found mon.c 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa480cd410 con 0x7faa480c9690 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:30.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa586c0640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa586c0640 10 monclient: _renew_subs 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa586c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa480cb3d0 con 0x7faa480c9690 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa586c0640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa586c0640 10 monclient: _renew_subs 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa586c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa48032f00 con 0x7faa480c9690 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.033+0000 7faa586c0640 1 librados: init done 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.036+0000 7faa586c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.036+0000 7faa586c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.036+0000 7faa586c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:30.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.036+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa48007bb0 con 0x7faa480c9690 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.037+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa5004f400 con 0x7faa480c9690 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.037+0000 7faa46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.037+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7faa50075710 con 0x7faa480c9690 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.037+0000 7faa46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.037+0000 7faa46ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.037+0000 7faa46ffd640 10 monclient: dump: 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:30.042 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.037+0000 7faa47fff640 4 set_mon_vals no callback set 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.038+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7faa50067bf0 con 0x7faa480c9690 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.039+0000 7faa5645c640 10 monclient: get_auth_request con 0x7faa3406dce0 auth_method 0 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.040+0000 7faa46ffd640 10 client.4481.objecter ms_handle_connect 0x7faa3406dce0 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.040+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7faa500ff460 con 0x7faa480c9690 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.040+0000 7faa46ffd640 10 client.4481.objecter ms_dispatch 0x7faa50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.040+0000 7faa46ffd640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.040+0000 7faa46ffd640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:30.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.040+0000 7faa46ffd640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:41:30.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.045+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa500cad40 con 0x7faa480c9690 2024-03-29T11:41:30.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.045+0000 7faa46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:30.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.045+0000 7faa46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:30.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.369+0000 7faa586c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:30.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.369+0000 7faa586c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:30.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.369+0000 7faa586c0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:30.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.369+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa48005140 con 0x7faa480c9690 2024-03-29T11:41:30.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.371+0000 7faa46ffd640 1 -- 172.21.15.145:0/3308833504 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7faa500cfe60 con 0x7faa480c9690 2024-03-29T11:41:30.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.371+0000 7faa46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:30.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.371+0000 7faa46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:30.376 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:41:30.376 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:30.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 10 librados: watch_flush enter 2024-03-29T11:41:30.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 10 librados: watch_flush exit 2024-03-29T11:41:30.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 >> v1:172.21.15.145:6816/227372239 conn(0x7faa3406dce0 legacy=0x7faa340701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 10 monclient: shutdown 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 >> v1:172.21.15.145:6790/0 conn(0x7faa480c9690 legacy=0x7faa4816c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 shutdown_connections 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 >> 172.21.15.145:0/3308833504 conn(0x7faa48005c20 msgr2=0x7faa480c8d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 shutdown_connections 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 1 -- 172.21.15.145:0/3308833504 wait complete. 2024-03-29T11:41:30.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:30.375+0000 7faa586c0640 1 librados: shutdown 2024-03-29T11:41:30.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:37.203 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:37.354 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.350+0000 7fa19fc0f640 10 monclient: get_monmap_and_config 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.350+0000 7fa19fc0f640 10 monclient: build_initial_monmap 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.350+0000 7fa19fc0f640 10 monclient: monmap: 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:41:37.350637+0000 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:41:37.350637+0000 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:41:37.355 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.351+0000 7fa19fc0f640 1 Processor -- start 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.351+0000 7fa19fc0f640 1 -- start start 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.351+0000 7fa19fc0f640 10 monclient: init 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 10 monclient(hunting): picked mon.noname-c con 0x7fa1981457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 10 monclient(hunting): picked mon.noname-a con 0x7fa19813f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 10 monclient(hunting): picked mon.noname-b con 0x7fa1980a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1980aaef0 con 0x7fa19813f650 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa198145bd0 con 0x7fa1981457f0 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa198145eb0 con 0x7fa1980a9d70 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:37.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.352+0000 7fa19fc0f640 10 monclient(hunting): authenticate will time out at 1286.659472s 2024-03-29T11:41:37.357 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.355+0000 7fa19d9ab640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa1981457f0 0x7fa19813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:38775/0 (socket says 172.21.15.145:38775) 2024-03-29T11:41:37.357 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.356+0000 7fa19d9ab640 1 -- 172.21.15.145:0/3994125776 learned_addr learned my addr 172.21.15.145:0/3994125776 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:37.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.356+0000 7fa19c9a9640 1 -- 172.21.15.145:0/3994125776 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169499483 0 0) 0x7fa1980aaef0 con 0x7fa19813f650 2024-03-29T11:41:37.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.356+0000 7fa19c9a9640 10 monclient(hunting): _negotiate 2024-03-29T11:41:37.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.356+0000 7fa19c9a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:37.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.356+0000 7fa19c9a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:37.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.356+0000 7fa19c9a9640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:41:37.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.356+0000 7fa19c9a9640 1 -- 172.21.15.145:0/3994125776 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa184003520 con 0x7fa19813f650 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 1 -- 172.21.15.145:0/3994125776 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176156814 0 0) 0x7fa184003520 con 0x7fa19813f650 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 1 -- 172.21.15.145:0/3994125776 >> v1:172.21.15.145:6790/0 conn(0x7fa1980a9d70 legacy=0x7fa19814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 1 -- 172.21.15.145:0/3994125776 >> v1:172.21.15.203:6789/0 conn(0x7fa1981457f0 legacy=0x7fa19813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 10 monclient: _finish_hunting 0 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 1 monclient: found mon.noname-a 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 1 -- 172.21.15.145:0/3994125776 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa198146340 con 0x7fa19813f650 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 10 monclient: _finish_auth 0 2024-03-29T11:41:37.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 10 monclient: _check_auth_tickets 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 1 -- 172.21.15.145:0/3994125776 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa18c001690 con 0x7fa19813f650 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 10 monclient: dump: 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:37.360 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:37.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19fc0f640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:41:37.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19fc0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:37.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 1 -- 172.21.15.145:0/3994125776 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa18c0018b0 con 0x7fa19813f650 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19c9a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19fc0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.357+0000 7fa19fc0f640 4 set_mon_vals no callback set 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.358+0000 7fa19fc0f640 10 monclient: shutdown 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.358+0000 7fa19fc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.358+0000 7fa19fc0f640 1 -- 172.21.15.145:0/3994125776 >> v1:172.21.15.145:6789/0 conn(0x7fa19813f650 legacy=0x7fa1980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.359+0000 7fa19fc0f640 1 -- 172.21.15.145:0/3994125776 shutdown_connections 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.359+0000 7fa19fc0f640 1 -- 172.21.15.145:0/3994125776 >> 172.21.15.145:0/3994125776 conn(0x7fa198066a00 msgr2=0x7fa198066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.359+0000 7fa19fc0f640 1 -- 172.21.15.145:0/3994125776 shutdown_connections 2024-03-29T11:41:37.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.359+0000 7fa19fc0f640 1 -- 172.21.15.145:0/3994125776 wait complete. 2024-03-29T11:41:37.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.360+0000 7fa19fc0f640 10 monclient: build_initial_monmap 2024-03-29T11:41:37.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.360+0000 7fa19fc0f640 10 monclient: monmap: 2024-03-29T11:41:37.363 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.360+0000 7fa19fc0f640 1 librados: starting msgr at 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.360+0000 7fa19fc0f640 1 librados: starting objecter 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.361+0000 7fa19fc0f640 1 Processor -- start 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.361+0000 7fa19fc0f640 1 -- start start 2024-03-29T11:41:37.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.361+0000 7fa19fc0f640 1 librados: setting wanted keys 2024-03-29T11:41:37.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.361+0000 7fa19fc0f640 1 librados: calling monclient init 2024-03-29T11:41:37.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.361+0000 7fa19fc0f640 10 monclient: init 2024-03-29T11:41:37.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.361+0000 7fa19fc0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:37.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.361+0000 7fa19fc0f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:41:37.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.362+0000 7fa19fc0f640 10 monclient(hunting): picked mon.noname-a con 0x7fa1980a9d70 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:37.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.362+0000 7fa19fc0f640 10 monclient(hunting): picked mon.noname-c con 0x7fa19813f650 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:37.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.362+0000 7fa19fc0f640 10 monclient(hunting): picked mon.noname-b con 0x7fa1981457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:37.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.362+0000 7fa19fc0f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa198137100 con 0x7fa1980a9d70 2024-03-29T11:41:37.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.362+0000 7fa19fc0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa19806eb40 con 0x7fa1981457f0 2024-03-29T11:41:37.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.362+0000 7fa19fc0f640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1981dbbc0 con 0x7fa19813f650 2024-03-29T11:41:37.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.362+0000 7fa19fc0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.362+0000 7fa19fc0f640 10 monclient(hunting): authenticate will time out at 1286.669095s 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.364+0000 7fa19d1aa640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fa19813f650 0x7fa19813fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:46592/0 (socket says 172.21.15.145:46592) 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.364+0000 7fa19d1aa640 1 -- 172.21.15.145:0/1802910936 learned_addr learned my addr 172.21.15.145:0/1802910936 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 client.?.objecter ms_handle_connect 0x7fa19813f650 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 client.?.objecter ms_handle_connect 0x7fa1981457f0 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354574877 0 0) 0x7fa1981dbbc0 con 0x7fa19813f650 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c004eb0 con 0x7fa19813f650 2024-03-29T11:41:37.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073531055 0 0) 0x7fa17c004eb0 con 0x7fa19813f650 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1981dbbc0 con 0x7fa19813f650 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576540086 0 0) 0x7fa19806eb40 con 0x7fa1981457f0 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa17c004eb0 con 0x7fa1981457f0 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa18c003850 con 0x7fa19813f650 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): dump: 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:37.373 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.365+0000 7fa18a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.366+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1825489712 0 0) 0x7fa1981dbbc0 con 0x7fa19813f650 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.366+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 >> v1:172.21.15.203:6789/0 conn(0x7fa1981457f0 legacy=0x7fa1981e6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.366+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 >> v1:172.21.15.145:6789/0 conn(0x7fa1980a9d70 legacy=0x7fa1981dc0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.368+0000 7fa18a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa18a7fc640 1 monclient: found mon.c 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa18a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:37.374 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa18a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:37.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1981db730 con 0x7fa19813f650 2024-03-29T11:41:37.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa18a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:37.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa18a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:37.375 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa18a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa19fc0f640 5 monclient: authenticate success, global_id 4487 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa19fc0f640 10 monclient: _renew_subs 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa19fc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa198137320 con 0x7fa19813f650 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa19fc0f640 10 client.4487.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa19fc0f640 10 monclient: _renew_subs 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa19fc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa198144b30 con 0x7fa19813f650 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.369+0000 7fa19fc0f640 1 librados: init done 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.371+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa18c0039b0 con 0x7fa19813f650 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.371+0000 7fa18a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.371+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa18c001c40 con 0x7fa19813f650 2024-03-29T11:41:37.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.371+0000 7fa18bfff640 4 set_mon_vals no callback set 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.373+0000 7fa18a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.373+0000 7fa18a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.373+0000 7fa18a7fc640 10 monclient: dump: 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.373+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fa18c003e00 con 0x7fa19813f650 2024-03-29T11:41:37.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.373+0000 7fa19fc0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.374+0000 7fa19fc0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.374+0000 7fa19fc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.374+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa180005080 con 0x7fa19813f650 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.374+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fa18c084d20 con 0x7fa19813f650 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.374+0000 7fa18a7fc640 10 client.4487.objecter ms_dispatch 0x7fa198000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.374+0000 7fa18a7fc640 3 client.4487.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.374+0000 7fa18a7fc640 3 client.4487.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.374+0000 7fa19d9ab640 10 monclient: get_auth_request con 0x7fa17c06de50 auth_method 0 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.375+0000 7fa18a7fc640 20 client.4487.objecter dump_active .. 0 homeless 2024-03-29T11:41:37.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.375+0000 7fa18a7fc640 10 client.4487.objecter ms_handle_connect 0x7fa17c06de50 2024-03-29T11:41:37.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.383+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa18c050430 con 0x7fa19813f650 2024-03-29T11:41:37.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.383+0000 7fa18a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:37.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.383+0000 7fa18a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:37.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.702+0000 7fa19fc0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:37.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.702+0000 7fa19fc0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:37.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.702+0000 7fa19fc0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:41:37.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.702+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa180005330 con 0x7fa19813f650 2024-03-29T11:41:37.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.703+0000 7fa18a7fc640 1 -- 172.21.15.145:0/1802910936 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fa18c052530 con 0x7fa19813f650 2024-03-29T11:41:37.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.703+0000 7fa18a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:37.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.703+0000 7fa18a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:37.706 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:41:37.706 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:37.709 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.707+0000 7fa19fc0f640 10 librados: watch_flush enter 2024-03-29T11:41:37.709 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.707+0000 7fa19fc0f640 10 librados: watch_flush exit 2024-03-29T11:41:37.709 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.707+0000 7fa19fc0f640 20 client.4487.objecter shutdown clearing up homeless session... 2024-03-29T11:41:37.709 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.707+0000 7fa19fc0f640 10 client.4487.objecter successfully canceled tick 2024-03-29T11:41:37.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.708+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 >> v1:172.21.15.145:6816/227372239 conn(0x7fa17c06de50 legacy=0x7fa17c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:37.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.708+0000 7fa19fc0f640 10 monclient: shutdown 2024-03-29T11:41:37.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.708+0000 7fa19fc0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:37.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.708+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 >> v1:172.21.15.145:6790/0 conn(0x7fa19813f650 legacy=0x7fa19813fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:37.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.709+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 shutdown_connections 2024-03-29T11:41:37.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.709+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 >> 172.21.15.145:0/1802910936 conn(0x7fa198066a00 msgr2=0x7fa19813b420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:37.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.709+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 shutdown_connections 2024-03-29T11:41:37.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.709+0000 7fa19fc0f640 1 -- 172.21.15.145:0/1802910936 wait complete. 2024-03-29T11:41:37.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:37.709+0000 7fa19fc0f640 1 librados: shutdown 2024-03-29T11:41:37.727 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:44.535 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:44.680 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.677+0000 7f3dbdd72640 10 monclient: get_monmap_and_config 2024-03-29T11:41:44.680 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.677+0000 7f3dbdd72640 10 monclient: build_initial_monmap 2024-03-29T11:41:44.680 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.677+0000 7f3dbdd72640 10 monclient: monmap: 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:41:44.678684+0000 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:41:44.678684+0000 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:41:44.681 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:44.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.678+0000 7f3dbdd72640 1 Processor -- start 2024-03-29T11:41:44.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.678+0000 7f3dbdd72640 1 -- start start 2024-03-29T11:41:44.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.678+0000 7f3dbdd72640 10 monclient: init 2024-03-29T11:41:44.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:44.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:41:44.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 10 monclient(hunting): picked mon.noname-c con 0x7f3db81457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:44.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 10 monclient(hunting): picked mon.noname-a con 0x7f3db813f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:44.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 10 monclient(hunting): picked mon.noname-b con 0x7f3db80a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:44.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db80aaef0 con 0x7f3db813f650 2024-03-29T11:41:44.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db8145bd0 con 0x7f3db81457f0 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3db77fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f3db81457f0 0x7f3db813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:57840/0 (socket says 172.21.15.145:57840) 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3db77fe640 1 -- 172.21.15.145:0/2270407274 learned_addr learned my addr 172.21.15.145:0/2270407274 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 1 -- 172.21.15.145:0/2270407274 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db8145eb0 con 0x7f3db80a9d70 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3dbdd72640 10 monclient(hunting): authenticate will time out at 1293.987767s 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367471995 0 0) 0x7f3db8145bd0 con 0x7f3db81457f0 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3db67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3db67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:44.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.679+0000 7f3db67fc640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da0003520 con 0x7f3db81457f0 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046096055 0 0) 0x7f3db8145eb0 con 0x7f3db80a9d70 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db8145bd0 con 0x7f3db80a9d70 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2285549874 0 0) 0x7f3da0003520 con 0x7f3db81457f0 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 >> v1:172.21.15.145:6790/0 conn(0x7f3db80a9d70 legacy=0x7f3db814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 >> v1:172.21.15.145:6789/0 conn(0x7f3db813f650 legacy=0x7f3db80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 monclient: found mon.noname-c 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db8146340 con 0x7f3db81457f0 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: _finish_auth 0 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3da8001690 con 0x7f3db81457f0 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3dbdd72640 5 monclient: authenticate success, global_id 4468 2024-03-29T11:41:44.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3dbdd72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: dump: 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:44.691 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3dbdd72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3da80018b0 con 0x7f3db81457f0 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 1 -- 172.21.15.145:0/2270407274 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3da8001c40 con 0x7f3db81457f0 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.680+0000 7f3db67fc640 10 monclient: dump: 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:44.692 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.681+0000 7f3dbdd72640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.681+0000 7f3dbdd72640 4 set_mon_vals no callback set 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.682+0000 7f3dbdd72640 10 monclient: shutdown 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.682+0000 7f3dbdd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.682+0000 7f3dbdd72640 1 -- 172.21.15.145:0/2270407274 >> v1:172.21.15.203:6789/0 conn(0x7f3db81457f0 legacy=0x7f3db813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.682+0000 7f3dbdd72640 1 -- 172.21.15.145:0/2270407274 shutdown_connections 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.682+0000 7f3dbdd72640 1 -- 172.21.15.145:0/2270407274 >> 172.21.15.145:0/2270407274 conn(0x7f3db8066a00 msgr2=0x7f3db8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.682+0000 7f3dbdd72640 1 -- 172.21.15.145:0/2270407274 shutdown_connections 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.683+0000 7f3dbdd72640 1 -- 172.21.15.145:0/2270407274 wait complete. 2024-03-29T11:41:44.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.683+0000 7f3dbdd72640 10 monclient: build_initial_monmap 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.683+0000 7f3dbdd72640 10 monclient: monmap: 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:41:44.694 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:44.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.684+0000 7f3dbdd72640 1 librados: starting msgr at 2024-03-29T11:41:44.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.684+0000 7f3dbdd72640 1 librados: starting objecter 2024-03-29T11:41:44.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.684+0000 7f3dbdd72640 1 Processor -- start 2024-03-29T11:41:44.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.685+0000 7f3dbdd72640 1 -- start start 2024-03-29T11:41:44.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.685+0000 7f3dbdd72640 1 librados: setting wanted keys 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.685+0000 7f3dbdd72640 1 librados: calling monclient init 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.685+0000 7f3dbdd72640 10 monclient: init 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.685+0000 7f3dbdd72640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.685+0000 7f3dbdd72640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.685+0000 7f3dbdd72640 10 monclient(hunting): picked mon.noname-c con 0x7f3db80a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.685+0000 7f3dbdd72640 10 monclient(hunting): picked mon.noname-a con 0x7f3db813f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3dbdd72640 10 monclient(hunting): picked mon.noname-b con 0x7f3db81457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3dbdd72640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db81dfd80 con 0x7f3db813f650 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3dbdd72640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db806f3d0 con 0x7f3db81457f0 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3dbdd72640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3db806de90 con 0x7f3db80a9d70 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3dbdd72640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3dbdd72640 10 monclient(hunting): authenticate will time out at 1293.994208s 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3db6ffd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f3db813f650 0x7f3db81def60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:45838/0 (socket says 172.21.15.145:45838) 2024-03-29T11:41:44.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3db77fe640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f3db80a9d70 0x7f3db806f7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:47838/0 (socket says 172.21.15.145:47838) 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3db77fe640 1 -- 172.21.15.145:0/1226583009 learned_addr learned my addr 172.21.15.145:0/1226583009 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 client.?.objecter ms_handle_connect 0x7f3db80a9d70 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 client.?.objecter ms_handle_connect 0x7f3db813f650 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102086419 0 0) 0x7f3db806de90 con 0x7f3db80a9d70 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3da4004eb0 con 0x7f3db80a9d70 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2771769990 0 0) 0x7f3db81dfd80 con 0x7f3db813f650 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:44.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.686+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db806de90 con 0x7f3db813f650 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3050989612 0 0) 0x7f3db806de90 con 0x7f3db813f650 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3db81dfd80 con 0x7f3db813f650 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3dac0018b0 con 0x7f3db813f650 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 monclient(hunting): dump: 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:44.700 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 client.?.objecter ms_handle_connect 0x7f3db81457f0 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 801737794 0 0) 0x7f3db81dfd80 con 0x7f3db813f650 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 >> v1:172.21.15.145:6790/0 conn(0x7f3db80a9d70 legacy=0x7f3db806f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 >> v1:172.21.15.203:6789/0 conn(0x7f3db81457f0 legacy=0x7f3db81df670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 1 monclient: found mon.a 2024-03-29T11:41:44.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db8137d20 con 0x7f3db813f650 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 monclient: _finish_auth 0 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3d9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3dbdd72640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3dbdd72640 10 monclient: _renew_subs 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3dbdd72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db81423b0 con 0x7f3db813f650 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3dbdd72640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3dbdd72640 10 monclient: _renew_subs 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3dbdd72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:44.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db8144f70 con 0x7f3db813f650 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.687+0000 7f3dbdd72640 1 librados: init done 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.688+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3dac003610 con 0x7f3db813f650 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.688+0000 7f3d9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.688+0000 7f3db5ffb640 4 set_mon_vals no callback set 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.688+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f3dac0039e0 con 0x7f3db813f650 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.688+0000 7f3d9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.688+0000 7f3d9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.688+0000 7f3d9ffff640 10 monclient: dump: 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:44.703 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.689+0000 7f3dbdd72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.689+0000 7f3dbdd72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.689+0000 7f3dbdd72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.689+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d98005080 con 0x7f3db813f650 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.691+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f3dac003bc0 con 0x7f3db813f650 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.691+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f3dac084a30 con 0x7f3db813f650 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.691+0000 7f3d9ffff640 10 client.4467.objecter ms_dispatch 0x7f3db8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:44.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.691+0000 7f3d9ffff640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:44.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.691+0000 7f3d9ffff640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:44.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.692+0000 7f3db77fe640 10 monclient: get_auth_request con 0x7f3da406dce0 auth_method 0 2024-03-29T11:41:44.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.692+0000 7f3d9ffff640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T11:41:44.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.692+0000 7f3d9ffff640 10 client.4467.objecter ms_handle_connect 0x7f3da406dce0 2024-03-29T11:41:44.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.700+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3dac050370 con 0x7f3db813f650 2024-03-29T11:41:44.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.700+0000 7f3d9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:44.705 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:44.700+0000 7f3d9ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:45.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.053+0000 7f3dbdd72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:45.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.053+0000 7f3dbdd72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:45.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.053+0000 7f3dbdd72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:45.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.053+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d98005330 con 0x7f3db813f650 2024-03-29T11:41:45.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.054+0000 7f3d9ffff640 1 -- 172.21.15.145:0/1226583009 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f3dac0522a0 con 0x7f3db813f650 2024-03-29T11:41:45.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.054+0000 7f3d9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:45.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.054+0000 7f3d9ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:45.058 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:41:45.058 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:45.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 10 librados: watch_flush enter 2024-03-29T11:41:45.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 10 librados: watch_flush exit 2024-03-29T11:41:45.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T11:41:45.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 10 client.4467.objecter successfully canceled tick 2024-03-29T11:41:45.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 >> v1:172.21.15.145:6816/227372239 conn(0x7f3da406dce0 legacy=0x7f3da40701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:45.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 10 monclient: shutdown 2024-03-29T11:41:45.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:45.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 >> v1:172.21.15.145:6789/0 conn(0x7f3db813f650 legacy=0x7f3db81def60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:45.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 shutdown_connections 2024-03-29T11:41:45.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.058+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 >> 172.21.15.145:0/1226583009 conn(0x7f3db8066a00 msgr2=0x7f3db8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:45.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.059+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 shutdown_connections 2024-03-29T11:41:45.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.059+0000 7f3dbdd72640 1 -- 172.21.15.145:0/1226583009 wait complete. 2024-03-29T11:41:45.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:45.059+0000 7f3dbdd72640 1 librados: shutdown 2024-03-29T11:41:45.076 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:51.883 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:52.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.024+0000 7f2330aa8640 10 monclient: get_monmap_and_config 2024-03-29T11:41:52.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.024+0000 7f2330aa8640 10 monclient: build_initial_monmap 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.024+0000 7f2330aa8640 10 monclient: monmap: 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:41:52.026078+0000 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:41:52.026078+0000 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:41:52.030 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:52.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.026+0000 7f2330aa8640 1 Processor -- start 2024-03-29T11:41:52.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.026+0000 7f2330aa8640 1 -- start start 2024-03-29T11:41:52.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.026+0000 7f2330aa8640 10 monclient: init 2024-03-29T11:41:52.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.027+0000 7f2330aa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:52.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.028+0000 7f2330aa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:52.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.028+0000 7f2330aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f232c0acf60 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:52.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.028+0000 7f2330aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f232c13ff40 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:52.036 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.028+0000 7f2330aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f232c1403f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:52.036 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.031+0000 7f2330aa8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c0b4310 con 0x7f232c13ff40 2024-03-29T11:41:52.036 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.031+0000 7f2330aa8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c140800 con 0x7f232c1403f0 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f2330aa8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c0aec00 con 0x7f232c0acf60 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f2329d9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f232c13ff40 0x7f232c148840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:52408/0 (socket says 172.21.15.145:52408) 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f2329d9b640 1 -- 172.21.15.145:0/2616868542 learned_addr learned my addr 172.21.15.145:0/2616868542 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f2330aa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f2330aa8640 10 monclient(hunting): authenticate will time out at 1301.340456s 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107376279 0 0) 0x7f232c0b4310 con 0x7f232c13ff40 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f232959a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f232959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f232959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.032+0000 7f232959a640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.033+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2314003520 con 0x7f232c13ff40 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.033+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114551172 0 0) 0x7f232c140800 con 0x7f232c1403f0 2024-03-29T11:41:52.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.033+0000 7f232959a640 10 monclient(hunting): _negotiate 2024-03-29T11:41:52.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.033+0000 7f232959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:52.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.033+0000 7f232959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:52.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.033+0000 7f232959a640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:41:52.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.033+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f232c0b4310 con 0x7f232c1403f0 2024-03-29T11:41:52.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.033+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2604920918 0 0) 0x7f2314003520 con 0x7f232c13ff40 2024-03-29T11:41:52.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.033+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 >> v1:172.21.15.145:6790/0 conn(0x7f232c0acf60 legacy=0x7f232c139b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:52.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.035+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 >> v1:172.21.15.203:6789/0 conn(0x7f232c1403f0 legacy=0x7f232c14bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:52.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.035+0000 7f232959a640 10 monclient: _finish_hunting 0 2024-03-29T11:41:52.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.035+0000 7f232959a640 1 monclient: found mon.noname-a 2024-03-29T11:41:52.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.035+0000 7f232959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:52.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.035+0000 7f232959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:41:52.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c0aeee0 con 0x7f232c13ff40 2024-03-29T11:41:52.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f232959a640 10 monclient: _finish_auth 0 2024-03-29T11:41:52.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f232959a640 10 monclient: _check_auth_tickets 2024-03-29T11:41:52.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f232959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f231c001690 con 0x7f232c13ff40 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f2330aa8640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f2330aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f232959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f232959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f232959a640 10 monclient: dump: 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:52.041 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:52.042 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:52.042 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:52.042 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:52.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.036+0000 7f2330aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:52.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.038+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f231c0018b0 con 0x7f232c13ff40 2024-03-29T11:41:52.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.038+0000 7f232959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:52.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.038+0000 7f232959a640 1 -- 172.21.15.145:0/2616868542 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f231c001c40 con 0x7f232c13ff40 2024-03-29T11:41:52.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.038+0000 7f232959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:52.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.038+0000 7f232959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:52.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.038+0000 7f232959a640 10 monclient: dump: 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.038+0000 7f2330aa8640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:52.044 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.038+0000 7f2330aa8640 4 set_mon_vals no callback set 2024-03-29T11:41:52.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.039+0000 7f2330aa8640 10 monclient: shutdown 2024-03-29T11:41:52.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.039+0000 7f2330aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:52.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.039+0000 7f2330aa8640 1 -- 172.21.15.145:0/2616868542 >> v1:172.21.15.145:6789/0 conn(0x7f232c13ff40 legacy=0x7f232c148840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:52.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.042+0000 7f2330aa8640 1 -- 172.21.15.145:0/2616868542 shutdown_connections 2024-03-29T11:41:52.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.042+0000 7f2330aa8640 1 -- 172.21.15.145:0/2616868542 >> 172.21.15.145:0/2616868542 conn(0x7f232c0b35e0 msgr2=0x7f232c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:52.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.042+0000 7f2330aa8640 1 -- 172.21.15.145:0/2616868542 shutdown_connections 2024-03-29T11:41:52.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.043+0000 7f2330aa8640 1 -- 172.21.15.145:0/2616868542 wait complete. 2024-03-29T11:41:52.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.044+0000 7f2330aa8640 10 monclient: build_initial_monmap 2024-03-29T11:41:52.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.045+0000 7f2330aa8640 10 monclient: monmap: 2024-03-29T11:41:52.049 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:52.049 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:52.049 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:41:52.049 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:41:52.049 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:52.049 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:52.049 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:52.050 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:41:52.050 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:41:52.050 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:52.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.046+0000 7f2330aa8640 1 librados: starting msgr at 2024-03-29T11:41:52.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.046+0000 7f2330aa8640 1 librados: starting objecter 2024-03-29T11:41:52.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.046+0000 7f2330aa8640 1 Processor -- start 2024-03-29T11:41:52.052 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.046+0000 7f2330aa8640 1 -- start start 2024-03-29T11:41:52.052 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.046+0000 7f2330aa8640 1 librados: setting wanted keys 2024-03-29T11:41:52.052 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.046+0000 7f2330aa8640 1 librados: calling monclient init 2024-03-29T11:41:52.052 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.046+0000 7f2330aa8640 10 monclient: init 2024-03-29T11:41:52.052 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.047+0000 7f2330aa8640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:52.052 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.047+0000 7f2330aa8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:41:52.052 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.047+0000 7f2330aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f232c0acf60 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:52.052 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.048+0000 7f2330aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f232c13ff40 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:52.052 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.048+0000 7f2330aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f232c1403f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:52.053 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.048+0000 7f2330aa8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c1c90b0 con 0x7f232c13ff40 2024-03-29T11:41:52.053 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.048+0000 7f2330aa8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c1d5460 con 0x7f232c1403f0 2024-03-29T11:41:52.053 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.048+0000 7f2330aa8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c1ca5a0 con 0x7f232c0acf60 2024-03-29T11:41:52.053 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.048+0000 7f2330aa8640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:52.053 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.048+0000 7f2330aa8640 10 monclient(hunting): authenticate will time out at 1301.356100s 2024-03-29T11:41:52.053 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.048+0000 7f2329d9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f232c13ff40 0x7f232c1c94d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:52580/0 (socket says 172.21.15.145:52580) 2024-03-29T11:41:52.053 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.048+0000 7f2329d9b640 1 -- 172.21.15.145:0/2762361711 learned_addr learned my addr 172.21.15.145:0/2762361711 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:52.058 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 10 client.?.objecter ms_handle_connect 0x7f232c13ff40 2024-03-29T11:41:52.058 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 10 client.?.objecter ms_handle_connect 0x7f232c0acf60 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685253997 0 0) 0x7f232c1c90b0 con 0x7f232c13ff40 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f230c004eb0 con 0x7f232c13ff40 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.051+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100287636 0 0) 0x7f232c1ca5a0 con 0x7f232c0acf60 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.052+0000 7f231affd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.052+0000 7f231affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:52.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.052+0000 7f231affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.052+0000 7f231affd640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.052+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f232c1c90b0 con 0x7f232c0acf60 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.052+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1991901461 0 0) 0x7f232c1c90b0 con 0x7f232c0acf60 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f232c1ca5a0 con 0x7f232c0acf60 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2320001510 con 0x7f232c0acf60 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 10 monclient(hunting): dump: 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:52.060 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1063232920 0 0) 0x7f230c004eb0 con 0x7f232c13ff40 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f232c1c90b0 con 0x7f232c13ff40 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f231c0015d0 con 0x7f232c13ff40 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:52.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.053+0000 7f231affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 10 monclient(hunting): dump: 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:52.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1264000642 0 0) 0x7f232c1c90b0 con 0x7f232c13ff40 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 >> v1:172.21.15.145:6790/0 conn(0x7f232c0acf60 legacy=0x7f232c1d58f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 >> v1:172.21.15.203:6789/0 conn(0x7f232c1403f0 legacy=0x7f232c1dfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 1 monclient: found mon.a 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c1d6000 con 0x7f232c13ff40 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 10 monclient: _finish_auth 0 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f231affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f2330aa8640 5 monclient: authenticate success, global_id 4473 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f2330aa8640 10 monclient: _renew_subs 2024-03-29T11:41:52.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f2330aa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.054+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232c13dd00 con 0x7f232c13ff40 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f2330aa8640 10 client.4473.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f2330aa8640 10 monclient: _renew_subs 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f2330aa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c1c9dc0 con 0x7f232c13ff40 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f2330aa8640 1 librados: init done 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f231c004a90 con 0x7f232c13ff40 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f231affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f231c004da0 con 0x7f232c13ff40 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f231affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f231affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:52.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f231affd640 10 monclient: dump: 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:52.065 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.055+0000 7f231bfff640 4 set_mon_vals no callback set 2024-03-29T11:41:52.066 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.057+0000 7f2330aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:52.066 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.058+0000 7f2330aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:52.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.058+0000 7f2330aa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:52.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.058+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2310005080 con 0x7f232c13ff40 2024-03-29T11:41:52.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.061+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f231c01ad70 con 0x7f232c13ff40 2024-03-29T11:41:52.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.062+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f231c084ac0 con 0x7f232c13ff40 2024-03-29T11:41:52.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.062+0000 7f231affd640 10 client.4473.objecter ms_dispatch 0x7f232c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:52.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.062+0000 7f231affd640 3 client.4473.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:52.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.062+0000 7f231affd640 3 client.4473.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:52.067 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.062+0000 7f231affd640 20 client.4473.objecter dump_active .. 0 homeless 2024-03-29T11:41:52.068 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.065+0000 7f232a59c640 10 monclient: get_auth_request con 0x7f230c06de70 auth_method 0 2024-03-29T11:41:52.069 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.066+0000 7f231affd640 10 client.4473.objecter ms_handle_connect 0x7f230c06de70 2024-03-29T11:41:52.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.070+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f231c0503a0 con 0x7f232c13ff40 2024-03-29T11:41:52.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.070+0000 7f231affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:52.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.070+0000 7f231affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:52.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.393+0000 7f2330aa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:52.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.393+0000 7f2330aa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:52.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.393+0000 7f2330aa8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:41:52.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.393+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2310005330 con 0x7f232c13ff40 2024-03-29T11:41:52.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.396+0000 7f231affd640 1 -- 172.21.15.145:0/2762361711 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f231c0522d0 con 0x7f232c13ff40 2024-03-29T11:41:52.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.396+0000 7f231affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:52.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.396+0000 7f231affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:52.403 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:41:52.404 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:52.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.404+0000 7f2330aa8640 10 librados: watch_flush enter 2024-03-29T11:41:52.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.404+0000 7f2330aa8640 10 librados: watch_flush exit 2024-03-29T11:41:52.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.404+0000 7f2330aa8640 20 client.4473.objecter shutdown clearing up homeless session... 2024-03-29T11:41:52.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.404+0000 7f2330aa8640 10 client.4473.objecter successfully canceled tick 2024-03-29T11:41:52.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.404+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 >> v1:172.21.15.145:6816/227372239 conn(0x7f230c06de70 legacy=0x7f230c070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:52.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.404+0000 7f2330aa8640 10 monclient: shutdown 2024-03-29T11:41:52.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.404+0000 7f2330aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:52.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.404+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 >> v1:172.21.15.145:6789/0 conn(0x7f232c13ff40 legacy=0x7f232c1c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:52.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.405+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 shutdown_connections 2024-03-29T11:41:52.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.405+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 >> 172.21.15.145:0/2762361711 conn(0x7f232c0b35e0 msgr2=0x7f232c0b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:52.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.408+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 shutdown_connections 2024-03-29T11:41:52.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.408+0000 7f2330aa8640 1 -- 172.21.15.145:0/2762361711 wait complete. 2024-03-29T11:41:52.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:52.408+0000 7f2330aa8640 1 librados: shutdown 2024-03-29T11:41:52.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:41:59.239 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:41:59.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.384+0000 7f5a38433640 10 monclient: get_monmap_and_config 2024-03-29T11:41:59.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.384+0000 7f5a38433640 10 monclient: build_initial_monmap 2024-03-29T11:41:59.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.384+0000 7f5a38433640 10 monclient: monmap: 2024-03-29T11:41:59.387 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:59.387 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:59.387 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:41:59.385588+0000 2024-03-29T11:41:59.388 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:41:59.385588+0000 2024-03-29T11:41:59.388 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:59.388 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:59.388 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:59.388 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:41:59.388 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:41:59.388 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:59.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.385+0000 7f5a38433640 1 Processor -- start 2024-03-29T11:41:59.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.385+0000 7f5a38433640 1 -- start start 2024-03-29T11:41:59.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.385+0000 7f5a38433640 10 monclient: init 2024-03-29T11:41:59.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:59.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:59.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 10 monclient(hunting): picked mon.noname-c con 0x7f5a301422b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:59.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 10 monclient(hunting): picked mon.noname-b con 0x7f5a30142690 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:59.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a359ce640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f5a30142690 0x7f5a300b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:40768/0 (socket says 172.21.15.145:40768) 2024-03-29T11:41:59.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 10 monclient(hunting): picked mon.noname-a con 0x7f5a300b17d0 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a359ce640 1 -- 172.21.15.145:0/2726217246 learned_addr learned my addr 172.21.15.145:0/2726217246 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 1 -- 172.21.15.145:0/2726217246 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a30142c80 con 0x7f5a300b17d0 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 1 -- 172.21.15.145:0/2726217246 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300aae40 con 0x7f5a301422b0 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 1 -- 172.21.15.145:0/2726217246 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a3013fbd0 con 0x7f5a30142690 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a38433640 10 monclient(hunting): authenticate will time out at 1308.694441s 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a351cd640 1 -- 172.21.15.145:0/2726217246 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565411797 0 0) 0x7f5a3013fbd0 con 0x7f5a30142690 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a351cd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a351cd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a351cd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:59.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a351cd640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:41:59.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a351cd640 1 -- 172.21.15.145:0/2726217246 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a1c003520 con 0x7f5a30142690 2024-03-29T11:41:59.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a351cd640 1 -- 172.21.15.145:0/2726217246 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3565085360 0 0) 0x7f5a1c003520 con 0x7f5a30142690 2024-03-29T11:41:59.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a351cd640 1 -- 172.21.15.145:0/2726217246 >> v1:172.21.15.203:6789/0 conn(0x7f5a301422b0 legacy=0x7f5a3013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:59.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.386+0000 7f5a351cd640 1 -- 172.21.15.145:0/2726217246 >> v1:172.21.15.145:6789/0 conn(0x7f5a300b17d0 legacy=0x7f5a300b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:41:59.393 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:59.393 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 1 monclient: found mon.noname-b 2024-03-29T11:41:59.393 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:59.393 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:41:59.393 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 1 -- 172.21.15.145:0/2726217246 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a30151940 con 0x7f5a30142690 2024-03-29T11:41:59.393 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: _finish_auth 0 2024-03-29T11:41:59.393 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 1 -- 172.21.15.145:0/2726217246 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a20001690 con 0x7f5a30142690 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: dump: 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:59.394 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a38433640 5 monclient: authenticate success, global_id 4499 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a38433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 1 -- 172.21.15.145:0/2726217246 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a200017f0 con 0x7f5a30142690 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 1 -- 172.21.15.145:0/2726217246 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a20001c40 con 0x7f5a30142690 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a351cd640 10 monclient: dump: 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:59.395 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:59.396 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:59.396 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:59.396 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:59.396 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:59.396 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a38433640 10 monclient: get_monmap_and_config success 2024-03-29T11:41:59.396 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.390+0000 7f5a38433640 4 set_mon_vals no callback set 2024-03-29T11:41:59.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.391+0000 7f5a38433640 10 monclient: shutdown 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.391+0000 7f5a38433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.391+0000 7f5a38433640 1 -- 172.21.15.145:0/2726217246 >> v1:172.21.15.145:6790/0 conn(0x7f5a30142690 legacy=0x7f5a300b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.393+0000 7f5a38433640 1 -- 172.21.15.145:0/2726217246 shutdown_connections 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.393+0000 7f5a38433640 1 -- 172.21.15.145:0/2726217246 >> 172.21.15.145:0/2726217246 conn(0x7f5a30066a00 msgr2=0x7f5a30066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.393+0000 7f5a38433640 1 -- 172.21.15.145:0/2726217246 shutdown_connections 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.393+0000 7f5a38433640 1 -- 172.21.15.145:0/2726217246 wait complete. 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.394+0000 7f5a38433640 10 monclient: build_initial_monmap 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.394+0000 7f5a38433640 10 monclient: monmap: 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:41:59.398 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.394+0000 7f5a38433640 1 librados: starting msgr at 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.394+0000 7f5a38433640 1 librados: starting objecter 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.394+0000 7f5a38433640 1 Processor -- start 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.394+0000 7f5a38433640 1 -- start start 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.394+0000 7f5a38433640 1 librados: setting wanted keys 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.394+0000 7f5a38433640 1 librados: calling monclient init 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.394+0000 7f5a38433640 10 monclient: init 2024-03-29T11:41:59.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.395+0000 7f5a38433640 10 monclient: _reopen_session rank -1 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.395+0000 7f5a38433640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.395+0000 7f5a38433640 10 monclient(hunting): picked mon.noname-b con 0x7f5a300b17d0 addr v1:172.21.15.203:6789/0 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.395+0000 7f5a38433640 10 monclient(hunting): picked mon.noname-c con 0x7f5a301422b0 addr v1:172.21.15.145:6790/0 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.395+0000 7f5a359ce640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f5a301422b0 0x7f5a3012fdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:42934/0 (socket says 172.21.15.145:42934) 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.396+0000 7f5a359ce640 1 -- 172.21.15.145:0/2632444760 learned_addr learned my addr 172.21.15.145:0/2632444760 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.396+0000 7f5a38433640 10 monclient(hunting): picked mon.noname-a con 0x7f5a30142690 addr v1:172.21.15.145:6789/0 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.396+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300ac730 con 0x7f5a30142690 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.396+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300ac9a0 con 0x7f5a300b17d0 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.396+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a300acc80 con 0x7f5a301422b0 2024-03-29T11:41:59.405 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.396+0000 7f5a38433640 10 monclient(hunting): _renew_subs 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.396+0000 7f5a38433640 10 monclient(hunting): authenticate will time out at 1308.704732s 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.396+0000 7f5a26ffd640 10 client.?.objecter ms_handle_connect 0x7f5a300b17d0 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2667096797 0 0) 0x7f5a300ac9a0 con 0x7f5a300b17d0 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a14004eb0 con 0x7f5a300b17d0 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 client.?.objecter ms_handle_connect 0x7f5a301422b0 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:59.406 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 client.?.objecter ms_handle_connect 0x7f5a30142690 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865236564 0 0) 0x7f5a300acc80 con 0x7f5a301422b0 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.397+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a300ac9a0 con 0x7f5a301422b0 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3749757068 0 0) 0x7f5a300ac730 con 0x7f5a30142690 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:41:59.407 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a300acc80 con 0x7f5a30142690 2024-03-29T11:41:59.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1851756753 0 0) 0x7f5a14004eb0 con 0x7f5a300b17d0 2024-03-29T11:41:59.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a300ac730 con 0x7f5a300b17d0 2024-03-29T11:41:59.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a2c0018b0 con 0x7f5a300b17d0 2024-03-29T11:41:59.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:59.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:41:59.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:59.408 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:59.408 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:59.408 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3612832184 0 0) 0x7f5a300acc80 con 0x7f5a30142690 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.398+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a14004eb0 con 0x7f5a30142690 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a28001890 con 0x7f5a30142690 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:59.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 10 monclient(hunting): dump: 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1980702711 0 0) 0x7f5a300ac730 con 0x7f5a300b17d0 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 >> v1:172.21.15.145:6790/0 conn(0x7f5a301422b0 legacy=0x7f5a3012fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:59.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 >> v1:172.21.15.145:6789/0 conn(0x7f5a30142690 legacy=0x7f5a301304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 1 monclient: found mon.b 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a300abbf0 con 0x7f5a300b17d0 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 10 monclient: _finish_auth 0 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a38433640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a38433640 10 monclient: _renew_subs 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a38433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a30132660 con 0x7f5a300b17d0 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a38433640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:41:59.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a38433640 10 monclient: _renew_subs 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a38433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a30132230 con 0x7f5a300b17d0 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a38433640 1 librados: init done 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a2c003610 con 0x7f5a300b17d0 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.399+0000 7f5a26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.400+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a2c0039e0 con 0x7f5a300b17d0 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.400+0000 7f5a26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.400+0000 7f5a26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.400+0000 7f5a26ffd640 10 monclient: dump: 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:41:59.412 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:41:59.413 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:41:59.413 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:41:59.413 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:41:59.413 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:41:59.413 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.400+0000 7f5a349cc640 4 set_mon_vals no callback set 2024-03-29T11:41:59.413 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.401+0000 7f5a38433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:59.413 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.401+0000 7f5a38433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:59.413 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.401+0000 7f5a38433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:59.413 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.401+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a18005080 con 0x7f5a300b17d0 2024-03-29T11:41:59.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.402+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f5a2c003b40 con 0x7f5a300b17d0 2024-03-29T11:41:59.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.403+0000 7f5a359ce640 10 monclient: get_auth_request con 0x7f5a1406dd30 auth_method 0 2024-03-29T11:41:59.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.403+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f5a2c0837e0 con 0x7f5a300b17d0 2024-03-29T11:41:59.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.403+0000 7f5a26ffd640 10 client.4477.objecter ms_dispatch 0x7f5a30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:41:59.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.403+0000 7f5a26ffd640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:41:59.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.403+0000 7f5a26ffd640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:41:59.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.403+0000 7f5a26ffd640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T11:41:59.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.403+0000 7f5a26ffd640 10 client.4477.objecter ms_handle_connect 0x7f5a1406dd30 2024-03-29T11:41:59.418 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.411+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a2c04f0c0 con 0x7f5a300b17d0 2024-03-29T11:41:59.418 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.411+0000 7f5a26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:41:59.418 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.411+0000 7f5a26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:41:59.708 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.705+0000 7f5a38433640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:59.708 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.705+0000 7f5a38433640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:59.708 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.705+0000 7f5a38433640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:41:59.709 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.705+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a18005330 con 0x7f5a300b17d0 2024-03-29T11:41:59.709 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.706+0000 7f5a26ffd640 1 -- 172.21.15.145:0/2632444760 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f5a2c050ff0 con 0x7f5a300b17d0 2024-03-29T11:41:59.709 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.706+0000 7f5a26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:41:59.709 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.706+0000 7f5a26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:41:59.709 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:41:59.710 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:41:59.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.710+0000 7f5a38433640 10 librados: watch_flush enter 2024-03-29T11:41:59.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.710+0000 7f5a38433640 10 librados: watch_flush exit 2024-03-29T11:41:59.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.710+0000 7f5a38433640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T11:41:59.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.710+0000 7f5a38433640 10 client.4477.objecter successfully canceled tick 2024-03-29T11:41:59.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.711+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 >> v1:172.21.15.145:6816/227372239 conn(0x7f5a1406dd30 legacy=0x7f5a140701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:59.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.711+0000 7f5a38433640 10 monclient: shutdown 2024-03-29T11:41:59.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.711+0000 7f5a38433640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:41:59.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.711+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 >> v1:172.21.15.203:6789/0 conn(0x7f5a300b17d0 legacy=0x7f5a30129cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:41:59.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.711+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 shutdown_connections 2024-03-29T11:41:59.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.711+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 >> 172.21.15.145:0/2632444760 conn(0x7f5a30066a00 msgr2=0x7f5a300b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:41:59.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.711+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 shutdown_connections 2024-03-29T11:41:59.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.712+0000 7f5a38433640 1 -- 172.21.15.145:0/2632444760 wait complete. 2024-03-29T11:41:59.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:41:59.712+0000 7f5a38433640 1 librados: shutdown 2024-03-29T11:41:59.728 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:06.535 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.683+0000 7ff7e740d640 10 monclient: get_monmap_and_config 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.683+0000 7ff7e740d640 10 monclient: build_initial_monmap 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.683+0000 7ff7e740d640 10 monclient: monmap: 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:42:06.685067+0000 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:42:06.685067+0000 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:42:06.687 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:42:06.688 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:06.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.685+0000 7ff7e740d640 1 Processor -- start 2024-03-29T11:42:06.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.685+0000 7ff7e740d640 1 -- start start 2024-03-29T11:42:06.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.685+0000 7ff7e740d640 10 monclient: init 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.686+0000 7ff7e740d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.686+0000 7ff7e740d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.686+0000 7ff7e740d640 10 monclient(hunting): picked mon.noname-a con 0x7ff7e00b0770 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.686+0000 7ff7e740d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7e014a700 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.688+0000 7ff7e740d640 10 monclient(hunting): picked mon.noname-c con 0x7ff7e00ac5a0 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.688+0000 7ff7e740d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e00aba70 con 0x7ff7e00b0770 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.688+0000 7ff7e740d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e00aecb0 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.688+0000 7ff7e740d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e00af150 con 0x7ff7e014a700 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.688+0000 7ff7e740d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:06.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.688+0000 7ff7e740d640 10 monclient(hunting): authenticate will time out at 1315.996311s 2024-03-29T11:42:06.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.689+0000 7ff7e49a8640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7ff7e014a700 0x7ff7e00abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:55743/0 (socket says 172.21.15.145:55743) 2024-03-29T11:42:06.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.689+0000 7ff7e49a8640 1 -- 172.21.15.145:0/107743888 learned_addr learned my addr 172.21.15.145:0/107743888 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:06.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.689+0000 7ff7d7fff640 1 -- 172.21.15.145:0/107743888 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498183274 0 0) 0x7ff7e00af150 con 0x7ff7e014a700 2024-03-29T11:42:06.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.689+0000 7ff7d7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:06.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.689+0000 7ff7d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:06.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.689+0000 7ff7d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:06.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.689+0000 7ff7d7fff640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:42:06.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.689+0000 7ff7d7fff640 1 -- 172.21.15.145:0/107743888 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7cc003520 con 0x7ff7e014a700 2024-03-29T11:42:06.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.690+0000 7ff7d7fff640 1 -- 172.21.15.145:0/107743888 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2557472977 0 0) 0x7ff7cc003520 con 0x7ff7e014a700 2024-03-29T11:42:06.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.690+0000 7ff7d7fff640 1 -- 172.21.15.145:0/107743888 >> v1:172.21.15.203:6789/0 conn(0x7ff7e00ac5a0 legacy=0x7ff7e0151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:06.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 1 -- 172.21.15.145:0/107743888 >> v1:172.21.15.145:6789/0 conn(0x7ff7e00b0770 legacy=0x7ff7e0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:42:06.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:06.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 1 monclient: found mon.noname-b 2024-03-29T11:42:06.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:06.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:42:06.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 1 -- 172.21.15.145:0/107743888 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e00ac9d0 con 0x7ff7e014a700 2024-03-29T11:42:06.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:06.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:06.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:06.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7e740d640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:42:06.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7e740d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:06.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 1 -- 172.21.15.145:0/107743888 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff7dc001690 con 0x7ff7e014a700 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 10 monclient: dump: 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:06.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7e740d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.691+0000 7ff7d7fff640 1 -- 172.21.15.145:0/107743888 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7dc001860 con 0x7ff7e014a700 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.692+0000 7ff7d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.692+0000 7ff7e740d640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.692+0000 7ff7e740d640 4 set_mon_vals no callback set 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.692+0000 7ff7e740d640 10 monclient: shutdown 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.692+0000 7ff7e740d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.692+0000 7ff7e740d640 1 -- 172.21.15.145:0/107743888 >> v1:172.21.15.145:6790/0 conn(0x7ff7e014a700 legacy=0x7ff7e00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.693+0000 7ff7e740d640 1 -- 172.21.15.145:0/107743888 shutdown_connections 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.693+0000 7ff7e740d640 1 -- 172.21.15.145:0/107743888 >> 172.21.15.145:0/107743888 conn(0x7ff7e006b660 msgr2=0x7ff7e006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.694+0000 7ff7e740d640 1 -- 172.21.15.145:0/107743888 shutdown_connections 2024-03-29T11:42:06.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.695+0000 7ff7e740d640 1 -- 172.21.15.145:0/107743888 wait complete. 2024-03-29T11:42:06.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.696+0000 7ff7e740d640 10 monclient: build_initial_monmap 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.696+0000 7ff7e740d640 10 monclient: monmap: 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.696+0000 7ff7e740d640 1 librados: starting msgr at 2024-03-29T11:42:06.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.696+0000 7ff7e740d640 1 librados: starting objecter 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.696+0000 7ff7e740d640 1 Processor -- start 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.696+0000 7ff7e740d640 1 -- start start 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.696+0000 7ff7e740d640 1 librados: setting wanted keys 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.696+0000 7ff7e740d640 1 librados: calling monclient init 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.696+0000 7ff7e740d640 10 monclient: init 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 10 monclient(hunting): picked mon.noname-a con 0x7ff7e00ac5a0 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 10 monclient(hunting): picked mon.noname-c con 0x7ff7e00b0770 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7e014a700 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e0122a30 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e0123730 con 0x7ff7e014a700 2024-03-29T11:42:06.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7e006ff30 con 0x7ff7e00b0770 2024-03-29T11:42:06.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:06.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.697+0000 7ff7e740d640 10 monclient(hunting): authenticate will time out at 1316.005556s 2024-03-29T11:42:06.708 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.704+0000 7ff7e51a9640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff7e00ac5a0 0x7ff7e0070580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:57745/0 (socket says 172.21.15.145:57745) 2024-03-29T11:42:06.708 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.704+0000 7ff7e51a9640 1 -- 172.21.15.145:0/3068236512 learned_addr learned my addr 172.21.15.145:0/3068236512 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:06.708 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 client.?.objecter ms_handle_connect 0x7ff7e00ac5a0 2024-03-29T11:42:06.708 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:06.708 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226440428 0 0) 0x7ff7e0122a30 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.708 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:06.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:06.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:06.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:42:06.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c4004eb0 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1379282426 0 0) 0x7ff7c4004eb0 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7e0122a30 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff7d8001ce0 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:06.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.705+0000 7ff7d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 10 client.?.objecter ms_handle_connect 0x7ff7e00b0770 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:06.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 655803304 0 0) 0x7ff7e0122a30 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 >> v1:172.21.15.145:6790/0 conn(0x7ff7e00b0770 legacy=0x7ff7e01e7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 >> v1:172.21.15.203:6789/0 conn(0x7ff7e014a700 legacy=0x7ff7e01ef350 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 1 monclient: found mon.a 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7e006fa50 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7e740d640 5 monclient: authenticate success, global_id 4479 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7e740d640 10 monclient: _renew_subs 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7e740d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:06.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.706+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7e006fd00 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7e740d640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:06.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7e740d640 10 monclient: _renew_subs 2024-03-29T11:42:06.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7e740d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7e0122e60 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7e740d640 1 librados: init done 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7d8002000 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff7d8003dc0 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7d5ffb640 10 monclient: dump: 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:06.715 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.707+0000 7ff7d77fe640 4 set_mon_vals no callback set 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.708+0000 7ff7e740d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.708+0000 7ff7e740d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.708+0000 7ff7e740d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.708+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c0005080 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.710+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7ff7d8003f20 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.711+0000 7ff7e49a8640 10 monclient: get_auth_request con 0x7ff7c406dbd0 auth_method 0 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.711+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7ff7d8084e60 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.711+0000 7ff7d5ffb640 10 client.4479.objecter ms_dispatch 0x7ff7e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.711+0000 7ff7d5ffb640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.711+0000 7ff7d5ffb640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:06.716 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.711+0000 7ff7d5ffb640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-29T11:42:06.717 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.711+0000 7ff7d5ffb640 10 client.4479.objecter ms_handle_connect 0x7ff7c406dbd0 2024-03-29T11:42:06.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.722+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7d8050740 con 0x7ff7e00ac5a0 2024-03-29T11:42:06.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.722+0000 7ff7d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:06.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:06.722+0000 7ff7d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:07.047 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.044+0000 7ff7e740d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:07.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.044+0000 7ff7e740d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:07.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.044+0000 7ff7e740d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:07.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.044+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7c0005330 con 0x7ff7e00ac5a0 2024-03-29T11:42:07.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.046+0000 7ff7d5ffb640 1 -- 172.21.15.145:0/3068236512 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7ff7d8052670 con 0x7ff7e00ac5a0 2024-03-29T11:42:07.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.046+0000 7ff7d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:07.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.046+0000 7ff7d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:07.050 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:42:07.050 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:07.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.052+0000 7ff7e740d640 10 librados: watch_flush enter 2024-03-29T11:42:07.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.052+0000 7ff7e740d640 10 librados: watch_flush exit 2024-03-29T11:42:07.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.052+0000 7ff7e740d640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-29T11:42:07.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.052+0000 7ff7e740d640 10 client.4479.objecter successfully canceled tick 2024-03-29T11:42:07.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.052+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 >> v1:172.21.15.145:6816/227372239 conn(0x7ff7c406dbd0 legacy=0x7ff7c4070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:07.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.052+0000 7ff7e740d640 10 monclient: shutdown 2024-03-29T11:42:07.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.052+0000 7ff7e740d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:07.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.052+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 >> v1:172.21.15.145:6789/0 conn(0x7ff7e00ac5a0 legacy=0x7ff7e0070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:07.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.053+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 shutdown_connections 2024-03-29T11:42:07.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.053+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 >> 172.21.15.145:0/3068236512 conn(0x7ff7e006b660 msgr2=0x7ff7e006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:07.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.053+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 shutdown_connections 2024-03-29T11:42:07.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.053+0000 7ff7e740d640 1 -- 172.21.15.145:0/3068236512 wait complete. 2024-03-29T11:42:07.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:07.053+0000 7ff7e740d640 1 librados: shutdown 2024-03-29T11:42:07.071 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:13.879 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:14.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.026+0000 7f131e803640 10 monclient: get_monmap_and_config 2024-03-29T11:42:14.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.026+0000 7f131e803640 10 monclient: build_initial_monmap 2024-03-29T11:42:14.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.026+0000 7f131e803640 10 monclient: monmap: 2024-03-29T11:42:14.029 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:14.029 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:14.030 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:42:14.027516+0000 2024-03-29T11:42:14.030 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:42:14.027516+0000 2024-03-29T11:42:14.030 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:14.030 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:14.030 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:14.030 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:42:14.030 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:42:14.030 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:14.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.027+0000 7f131e803640 1 Processor -- start 2024-03-29T11:42:14.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.027+0000 7f131e803640 1 -- start start 2024-03-29T11:42:14.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.027+0000 7f131e803640 10 monclient: init 2024-03-29T11:42:14.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:14.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:14.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 10 monclient(hunting): picked mon.noname-a con 0x7f131806ade0 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:14.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 10 monclient(hunting): picked mon.noname-c con 0x7f1318072090 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:14.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 10 monclient(hunting): picked mon.noname-b con 0x7f131814a110 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:14.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131806fdb0 con 0x7f131806ade0 2024-03-29T11:42:14.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1318144010 con 0x7f1318072090 2024-03-29T11:42:14.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f131806eb00 con 0x7f131814a110 2024-03-29T11:42:14.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:14.033 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.028+0000 7f131e803640 10 monclient(hunting): authenticate will time out at 1323.336962s 2024-03-29T11:42:14.035 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.032+0000 7f131d801640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f131806ade0 0x7f1318071980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:48193/0 (socket says 172.21.15.145:48193) 2024-03-29T11:42:14.036 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.032+0000 7f131d801640 1 -- 172.21.15.145:0/4095422876 learned_addr learned my addr 172.21.15.145:0/4095422876 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:14.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254549795 0 0) 0x7f131806eb00 con 0x7f131814a110 2024-03-29T11:42:14.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:14.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1300003520 con 0x7f131814a110 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147172840 0 0) 0x7f131806fdb0 con 0x7f131806ade0 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f131806eb00 con 0x7f131806ade0 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3176926793 0 0) 0x7f1318144010 con 0x7f1318072090 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:42:14.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f131806fdb0 con 0x7f1318072090 2024-03-29T11:42:14.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 236195308 0 0) 0x7f1300003520 con 0x7f131814a110 2024-03-29T11:42:14.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 >> v1:172.21.15.203:6789/0 conn(0x7f1318072090 legacy=0x7f1318149a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:14.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 >> v1:172.21.15.145:6789/0 conn(0x7f131806ade0 legacy=0x7f1318071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:14.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:14.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 monclient: found mon.noname-b 2024-03-29T11:42:14.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:14.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:42:14.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1318070090 con 0x7f131814a110 2024-03-29T11:42:14.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: _finish_auth 0 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f130c001690 con 0x7f131814a110 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: dump: 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:14.044 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f130c001870 con 0x7f131814a110 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 1 -- 172.21.15.145:0/4095422876 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f130c001690 con 0x7f131814a110 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.036+0000 7f1317fff640 10 monclient: dump: 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:14.045 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.037+0000 7f131e803640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.037+0000 7f131e803640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.037+0000 7f131e803640 4 set_mon_vals no callback set 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.038+0000 7f131e803640 10 monclient: shutdown 2024-03-29T11:42:14.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.038+0000 7f131e803640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:14.047 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.038+0000 7f131e803640 1 -- 172.21.15.145:0/4095422876 >> v1:172.21.15.145:6790/0 conn(0x7f131814a110 legacy=0x7f1318151ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:14.047 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.038+0000 7f131e803640 1 -- 172.21.15.145:0/4095422876 shutdown_connections 2024-03-29T11:42:14.047 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.038+0000 7f131e803640 1 -- 172.21.15.145:0/4095422876 >> 172.21.15.145:0/4095422876 conn(0x7f1318067420 msgr2=0x7f13180b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.040+0000 7f131e803640 1 -- 172.21.15.145:0/4095422876 shutdown_connections 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.040+0000 7f131e803640 1 -- 172.21.15.145:0/4095422876 wait complete. 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.041+0000 7f131e803640 10 monclient: build_initial_monmap 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.041+0000 7f131e803640 10 monclient: monmap: 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:14.048 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:14.049 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:42:14.049 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:42:14.049 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:14.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.041+0000 7f131e803640 1 librados: starting msgr at 2024-03-29T11:42:14.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.041+0000 7f131e803640 1 librados: starting objecter 2024-03-29T11:42:14.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.042+0000 7f131e803640 1 Processor -- start 2024-03-29T11:42:14.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.042+0000 7f131e803640 1 -- start start 2024-03-29T11:42:14.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.042+0000 7f131e803640 1 librados: setting wanted keys 2024-03-29T11:42:14.049 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.042+0000 7f131e803640 1 librados: calling monclient init 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.042+0000 7f131e803640 10 monclient: init 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.042+0000 7f131e803640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.042+0000 7f131e803640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.043+0000 7f131e803640 10 monclient(hunting): picked mon.noname-c con 0x7f131806ade0 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.044+0000 7f131e803640 10 monclient(hunting): picked mon.noname-b con 0x7f1318072090 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.044+0000 7f131e803640 10 monclient(hunting): picked mon.noname-a con 0x7f131814a110 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.044+0000 7f131e803640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13180ac700 con 0x7f131814a110 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.044+0000 7f131e803640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13180aaff0 con 0x7f1318072090 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.044+0000 7f131d000640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f1318072090 0x7f13180acb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:35278/0 (socket says 172.21.15.145:35278) 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.044+0000 7f131d000640 1 -- 172.21.15.145:0/1503339569 learned_addr learned my addr 172.21.15.145:0/1503339569 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.044+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1318130d30 con 0x7f131806ade0 2024-03-29T11:42:14.050 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.044+0000 7f131e803640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:14.051 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.044+0000 7f131e803640 10 monclient(hunting): authenticate will time out at 1323.352473s 2024-03-29T11:42:14.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.045+0000 7f1315ffb640 10 client.?.objecter ms_handle_connect 0x7f1318072090 2024-03-29T11:42:14.056 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.045+0000 7f1315ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.045+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800250321 0 0) 0x7f13180aaff0 con 0x7f1318072090 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.045+0000 7f1315ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.045+0000 7f1315ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.045+0000 7f1315ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.045+0000 7f1315ffb640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.045+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1304004eb0 con 0x7f1318072090 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2061065640 0 0) 0x7f1304004eb0 con 0x7f1318072090 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13180aaff0 con 0x7f1318072090 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 client.?.objecter ms_handle_connect 0x7f131806ade0 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1310001c90 con 0x7f1318072090 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:14.057 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167534413 0 0) 0x7f1318130d30 con 0x7f131806ade0 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:42:14.058 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.046+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1304004eb0 con 0x7f131806ade0 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.047+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 729602420 0 0) 0x7f1304004eb0 con 0x7f131806ade0 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.047+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1318130d30 con 0x7f131806ade0 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.048+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1308001250 con 0x7f131806ade0 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.048+0000 7f1315ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.048+0000 7f1315ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.048+0000 7f1315ffb640 10 monclient(hunting): dump: 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.059 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.048+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 419694070 0 0) 0x7f13180aaff0 con 0x7f1318072090 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.048+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 >> v1:172.21.15.145:6790/0 conn(0x7f131806ade0 legacy=0x7f131813d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.048+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 >> v1:172.21.15.145:6789/0 conn(0x7f131814a110 legacy=0x7f1318123cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f1315ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f1315ffb640 1 monclient: found mon.b 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f1315ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f1315ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13180ab700 con 0x7f1318072090 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f1315ffb640 10 monclient: _finish_auth 0 2024-03-29T11:42:14.060 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f1315ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f1315ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f131e803640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f131e803640 10 monclient: _renew_subs 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f131e803640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.049+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13180ac1e0 con 0x7f1318072090 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f131e803640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f131e803640 10 monclient: _renew_subs 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f131e803640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f131813e2b0 con 0x7f1318072090 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f131e803640 1 librados: init done 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13100020e0 con 0x7f1318072090 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f1315ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1310003c40 con 0x7f1318072090 2024-03-29T11:42:14.061 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f1315ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f1315ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f1315ffb640 10 monclient: dump: 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.050+0000 7f13177fe640 4 set_mon_vals no callback set 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.052+0000 7f131e803640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:14.062 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.052+0000 7f131e803640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:14.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.052+0000 7f131e803640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:14.063 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.052+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f8005080 con 0x7f1318072090 2024-03-29T11:42:14.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.057+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f1310003da0 con 0x7f1318072090 2024-03-29T11:42:14.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.058+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f1310084da0 con 0x7f1318072090 2024-03-29T11:42:14.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.059+0000 7f1315ffb640 10 client.4483.objecter ms_dispatch 0x7f1318000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:14.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.059+0000 7f1315ffb640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:14.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.059+0000 7f1315ffb640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:14.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.059+0000 7f131d801640 10 monclient: get_auth_request con 0x7f130406dfa0 auth_method 0 2024-03-29T11:42:14.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.059+0000 7f1315ffb640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-29T11:42:14.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.059+0000 7f1315ffb640 10 client.4483.objecter ms_handle_connect 0x7f130406dfa0 2024-03-29T11:42:14.064 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.061+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1310050680 con 0x7f1318072090 2024-03-29T11:42:14.065 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.061+0000 7f1315ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:14.065 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.061+0000 7f1315ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:14.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.388+0000 7f131e803640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:14.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.388+0000 7f131e803640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:14.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.388+0000 7f131e803640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:14.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.388+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12f8005330 con 0x7f1318072090 2024-03-29T11:42:14.394 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.391+0000 7f1315ffb640 1 -- 172.21.15.145:0/1503339569 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f13100525b0 con 0x7f1318072090 2024-03-29T11:42:14.394 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.391+0000 7f1315ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:14.394 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.391+0000 7f1315ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:14.394 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:42:14.394 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:14.396 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.393+0000 7f131e803640 10 librados: watch_flush enter 2024-03-29T11:42:14.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.394+0000 7f131e803640 10 librados: watch_flush exit 2024-03-29T11:42:14.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.394+0000 7f131e803640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-29T11:42:14.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.394+0000 7f131e803640 10 client.4483.objecter successfully canceled tick 2024-03-29T11:42:14.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.396+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 >> v1:172.21.15.145:6816/227372239 conn(0x7f130406dfa0 legacy=0x7f1304070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:14.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.396+0000 7f131e803640 10 monclient: shutdown 2024-03-29T11:42:14.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.396+0000 7f131e803640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:14.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.396+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 >> v1:172.21.15.203:6789/0 conn(0x7f1318072090 legacy=0x7f13180acb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:14.400 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.396+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 shutdown_connections 2024-03-29T11:42:14.400 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.396+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 >> 172.21.15.145:0/1503339569 conn(0x7f1318067420 msgr2=0x7f13180716c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:14.400 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.397+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 shutdown_connections 2024-03-29T11:42:14.400 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.397+0000 7f131e803640 1 -- 172.21.15.145:0/1503339569 wait complete. 2024-03-29T11:42:14.400 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:14.397+0000 7f131e803640 1 librados: shutdown 2024-03-29T11:42:14.417 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:21.225 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.373+0000 7f5496e0f640 10 monclient: get_monmap_and_config 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.373+0000 7f5496e0f640 10 monclient: build_initial_monmap 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.373+0000 7f5496e0f640 10 monclient: monmap: 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:42:21.373577+0000 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:42:21.373577+0000 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:42:21.376 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:42:21.377 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:21.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.375+0000 7f5496e0f640 1 Processor -- start 2024-03-29T11:42:21.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.375+0000 7f5496e0f640 1 -- start start 2024-03-29T11:42:21.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.375+0000 7f5496e0f640 10 monclient: init 2024-03-29T11:42:21.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:21.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:21.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f54900b0770 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:21.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f549014a700 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:21.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f54900ac5a0 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:21.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54900aba70 con 0x7f54900ac5a0 2024-03-29T11:42:21.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54900aecb0 con 0x7f549014a700 2024-03-29T11:42:21.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54900af150 con 0x7f54900b0770 2024-03-29T11:42:21.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:21.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.376+0000 7f5496e0f640 10 monclient(hunting): authenticate will time out at 1330.683583s 2024-03-29T11:42:21.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.380+0000 7f54953ac640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f54900ac5a0 0x7f5490151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:54761/0 (socket says 172.21.15.145:54761) 2024-03-29T11:42:21.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.380+0000 7f54953ac640 1 -- 172.21.15.145:0/2537645750 learned_addr learned my addr 172.21.15.145:0/2537645750 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:21.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.383+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131605888 0 0) 0x7f54900af150 con 0x7f54900b0770 2024-03-29T11:42:21.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.383+0000 7f548f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:21.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.383+0000 7f548f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:21.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.383+0000 7f548f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:21.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.383+0000 7f548f7fe640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:42:21.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.383+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5478003520 con 0x7f54900b0770 2024-03-29T11:42:21.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031606124 0 0) 0x7f54900aba70 con 0x7f54900ac5a0 2024-03-29T11:42:21.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:21.387 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:21.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:21.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:42:21.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54900af150 con 0x7f54900ac5a0 2024-03-29T11:42:21.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215203359 0 0) 0x7f54900aecb0 con 0x7f549014a700 2024-03-29T11:42:21.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:42:21.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:21.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:21.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:42:21.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54780039d0 con 0x7f549014a700 2024-03-29T11:42:21.389 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3629633788 0 0) 0x7f54900af150 con 0x7f54900ac5a0 2024-03-29T11:42:21.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 >> v1:172.21.15.145:6790/0 conn(0x7f54900b0770 legacy=0x7f5490149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:21.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 >> v1:172.21.15.203:6789/0 conn(0x7f549014a700 legacy=0x7f54900abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:21.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:42:21.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:42:21.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:21.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:42:21.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54900ac9d0 con 0x7f54900ac5a0 2024-03-29T11:42:21.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:42:21.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f54840015f0 con 0x7f54900ac5a0 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: dump: 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:21.391 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f5496e0f640 5 monclient: authenticate success, global_id 4485 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f5496e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54840017d0 con 0x7f54900ac5a0 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 1 -- 172.21.15.145:0/2537645750 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f54840015f0 con 0x7f54900ac5a0 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:21.392 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:21.393 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f548f7fe640 10 monclient: dump: 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:21.394 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:21.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f5496e0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:21.395 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.384+0000 7f5496e0f640 4 set_mon_vals no callback set 2024-03-29T11:42:21.396 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.385+0000 7f5496e0f640 10 monclient: shutdown 2024-03-29T11:42:21.396 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.385+0000 7f5496e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:21.396 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.385+0000 7f5496e0f640 1 -- 172.21.15.145:0/2537645750 >> v1:172.21.15.145:6789/0 conn(0x7f54900ac5a0 legacy=0x7f5490151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.385+0000 7f5496e0f640 1 -- 172.21.15.145:0/2537645750 shutdown_connections 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.385+0000 7f5496e0f640 1 -- 172.21.15.145:0/2537645750 >> 172.21.15.145:0/2537645750 conn(0x7f549006b660 msgr2=0x7f549006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.389+0000 7f5496e0f640 1 -- 172.21.15.145:0/2537645750 shutdown_connections 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.390+0000 7f5496e0f640 1 -- 172.21.15.145:0/2537645750 wait complete. 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.391+0000 7f5496e0f640 10 monclient: build_initial_monmap 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.391+0000 7f5496e0f640 10 monclient: monmap: 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:42:21.397 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.391+0000 7f5496e0f640 1 librados: starting msgr at 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.391+0000 7f5496e0f640 1 librados: starting objecter 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.391+0000 7f5496e0f640 1 Processor -- start 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.391+0000 7f5496e0f640 1 -- start start 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.391+0000 7f5496e0f640 1 librados: setting wanted keys 2024-03-29T11:42:21.398 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.391+0000 7f5496e0f640 1 librados: calling monclient init 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.391+0000 7f5496e0f640 10 monclient: init 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f54900ac5a0 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f54900b0770 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f549014a700 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5490121f00 con 0x7f54900b0770 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5490121a20 con 0x7f549014a700 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5490121bb0 con 0x7f54900ac5a0 2024-03-29T11:42:21.399 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:21.400 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.392+0000 7f5496e0f640 10 monclient(hunting): authenticate will time out at 1330.699833s 2024-03-29T11:42:21.401 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f5494bab640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f54900ac5a0 0x7f5490147ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:48177/0 (socket says 172.21.15.145:48177) 2024-03-29T11:42:21.401 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f5494bab640 1 -- 172.21.15.145:0/2221348707 learned_addr learned my addr 172.21.15.145:0/2221348707 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:21.401 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f548d7fa640 10 client.?.objecter ms_handle_connect 0x7f54900ac5a0 2024-03-29T11:42:21.401 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f548d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:21.401 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 781141337 0 0) 0x7f5490121bb0 con 0x7f54900ac5a0 2024-03-29T11:42:21.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f548d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:21.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f548d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:21.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f548d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:21.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f548d7fa640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:42:21.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.397+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f547c004eb0 con 0x7f54900ac5a0 2024-03-29T11:42:21.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.398+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3069211312 0 0) 0x7f547c004eb0 con 0x7f54900ac5a0 2024-03-29T11:42:21.402 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.398+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5490121bb0 con 0x7f54900ac5a0 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.398+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5480001990 con 0x7f54900ac5a0 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.398+0000 7f548d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.398+0000 7f548d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.398+0000 7f548d7fa640 10 monclient(hunting): dump: 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:21.403 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:21.404 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:21.404 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:21.404 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:21.404 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.398+0000 7f548d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:21.404 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.398+0000 7f548d7fa640 10 client.?.objecter ms_handle_connect 0x7f54900b0770 2024-03-29T11:42:21.404 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.398+0000 7f548d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:21.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 client.?.objecter ms_handle_connect 0x7f549014a700 2024-03-29T11:42:21.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:21.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2316980053 0 0) 0x7f5490121a20 con 0x7f549014a700 2024-03-29T11:42:21.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:42:21.408 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f547c004eb0 con 0x7f549014a700 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1718085481 0 0) 0x7f5490121bb0 con 0x7f54900ac5a0 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 >> v1:172.21.15.203:6789/0 conn(0x7f549014a700 legacy=0x7f54901e7840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 >> v1:172.21.15.145:6789/0 conn(0x7f54900b0770 legacy=0x7f5490122390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 1 monclient: found mon.c 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5490148700 con 0x7f54900ac5a0 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:42:21.409 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.400+0000 7f548d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.401+0000 7f5496e0f640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.401+0000 7f5496e0f640 10 monclient: _renew_subs 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.401+0000 7f5496e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.401+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5490148950 con 0x7f54900ac5a0 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.401+0000 7f5496e0f640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.401+0000 7f5496e0f640 10 monclient: _renew_subs 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.401+0000 7f5496e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.401+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5490122c80 con 0x7f54900ac5a0 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.401+0000 7f5496e0f640 1 librados: init done 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.402+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5480001cd0 con 0x7f54900ac5a0 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.402+0000 7f548d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.402+0000 7f548e7fc640 4 set_mon_vals no callback set 2024-03-29T11:42:21.410 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.403+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5480003a60 con 0x7f54900ac5a0 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.403+0000 7f548d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.403+0000 7f548d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.403+0000 7f548d7fa640 10 monclient: dump: 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.404+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f5480003bc0 con 0x7f54900ac5a0 2024-03-29T11:42:21.411 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.404+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f5480084ad0 con 0x7f54900ac5a0 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.404+0000 7f548d7fa640 10 client.4520.objecter ms_dispatch 0x7f5490000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.404+0000 7f548d7fa640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.404+0000 7f548d7fa640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.404+0000 7f548d7fa640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.404+0000 7f548ffff640 10 monclient: get_auth_request con 0x7f547c06dcc0 auth_method 0 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.404+0000 7f5496e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.404+0000 7f5496e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.405+0000 7f5496e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.405+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5470005080 con 0x7f54900ac5a0 2024-03-29T11:42:21.412 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.405+0000 7f548d7fa640 10 client.4520.objecter ms_handle_connect 0x7f547c06dcc0 2024-03-29T11:42:21.414 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.412+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54800503b0 con 0x7f54900ac5a0 2024-03-29T11:42:21.415 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.412+0000 7f548d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:21.415 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.412+0000 7f548d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:21.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.732+0000 7f5496e0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:21.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.732+0000 7f5496e0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:21.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.733+0000 7f5496e0f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:21.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.733+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5470005330 con 0x7f54900ac5a0 2024-03-29T11:42:21.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.733+0000 7f548d7fa640 1 -- 172.21.15.145:0/2221348707 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f54800522e0 con 0x7f54900ac5a0 2024-03-29T11:42:21.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.733+0000 7f548d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:21.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.733+0000 7f548d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:21.737 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:42:21.737 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:21.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.737+0000 7f5496e0f640 10 librados: watch_flush enter 2024-03-29T11:42:21.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.737+0000 7f5496e0f640 10 librados: watch_flush exit 2024-03-29T11:42:21.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.737+0000 7f5496e0f640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:42:21.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.737+0000 7f5496e0f640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:42:21.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.739+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 >> v1:172.21.15.145:6816/227372239 conn(0x7f547c06dcc0 legacy=0x7f547c070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:21.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.739+0000 7f5496e0f640 10 monclient: shutdown 2024-03-29T11:42:21.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.739+0000 7f5496e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:21.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.739+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 >> v1:172.21.15.145:6790/0 conn(0x7f54900ac5a0 legacy=0x7f5490147ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:21.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.740+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 shutdown_connections 2024-03-29T11:42:21.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.740+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 >> 172.21.15.145:0/2221348707 conn(0x7f549006b660 msgr2=0x7f549006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:21.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.741+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 shutdown_connections 2024-03-29T11:42:21.744 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.741+0000 7f5496e0f640 1 -- 172.21.15.145:0/2221348707 wait complete. 2024-03-29T11:42:21.744 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:21.741+0000 7f5496e0f640 1 librados: shutdown 2024-03-29T11:42:21.761 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:28.568 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:28.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.724+0000 7f859e8ff640 10 monclient: get_monmap_and_config 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.724+0000 7f859e8ff640 10 monclient: build_initial_monmap 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.724+0000 7f859e8ff640 10 monclient: monmap: 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:42:28.724705+0000 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:42:28.724705+0000 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:42:28.727 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:28.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.725+0000 7f859e8ff640 1 Processor -- start 2024-03-29T11:42:28.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.725+0000 7f859e8ff640 1 -- start start 2024-03-29T11:42:28.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.725+0000 7f859e8ff640 10 monclient: init 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 10 monclient(hunting): picked mon.noname-c con 0x7f85981457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 10 monclient(hunting): picked mon.noname-a con 0x7f859813f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 10 monclient(hunting): picked mon.noname-b con 0x7f85980a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980aaef0 con 0x7f859813f650 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8598145bd0 con 0x7f85981457f0 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8598145eb0 con 0x7f85980a9d70 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:28.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.726+0000 7f859e8ff640 10 monclient(hunting): authenticate will time out at 1338.033882s 2024-03-29T11:42:28.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f859ce9c640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f85980a9d70 0x7f859814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:52272/0 (socket says 172.21.15.145:52272) 2024-03-29T11:42:28.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f859ce9c640 1 -- 172.21.15.145:0/3832646577 learned_addr learned my addr 172.21.15.145:0/3832646577 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:28.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599223331 0 0) 0x7f8598145bd0 con 0x7f85981457f0 2024-03-29T11:42:28.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:28.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:28.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8580003520 con 0x7f85981457f0 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282321242 0 0) 0x7f85980aaef0 con 0x7f859813f650 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8598145bd0 con 0x7f859813f650 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192203140 0 0) 0x7f8598145eb0 con 0x7f85980a9d70 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:42:28.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.727+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85980aaef0 con 0x7f85980a9d70 2024-03-29T11:42:28.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3872081502 0 0) 0x7f8580003520 con 0x7f85981457f0 2024-03-29T11:42:28.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 >> v1:172.21.15.145:6790/0 conn(0x7f85980a9d70 legacy=0x7f859814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 >> v1:172.21.15.145:6789/0 conn(0x7f859813f650 legacy=0x7f85980b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 1 monclient: found mon.noname-c 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8598146340 con 0x7f85981457f0 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f85880015c0 con 0x7f85981457f0 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.728+0000 7f8596ffd640 10 monclient: dump: 2024-03-29T11:42:28.733 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.729+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8588001720 con 0x7f85981457f0 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.729+0000 7f8596ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.729+0000 7f8596ffd640 1 -- 172.21.15.145:0/3832646577 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8588001b70 con 0x7f85981457f0 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.729+0000 7f8596ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.734 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.729+0000 7f8596ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.729+0000 7f8596ffd640 10 monclient: dump: 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.729+0000 7f859e8ff640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.729+0000 7f859e8ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.729+0000 7f859e8ff640 4 set_mon_vals no callback set 2024-03-29T11:42:28.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.730+0000 7f859e8ff640 10 monclient: shutdown 2024-03-29T11:42:28.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.730+0000 7f859e8ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:28.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.730+0000 7f859e8ff640 1 -- 172.21.15.145:0/3832646577 >> v1:172.21.15.203:6789/0 conn(0x7f85981457f0 legacy=0x7f859813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.730+0000 7f859e8ff640 1 -- 172.21.15.145:0/3832646577 shutdown_connections 2024-03-29T11:42:28.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.730+0000 7f859e8ff640 1 -- 172.21.15.145:0/3832646577 >> 172.21.15.145:0/3832646577 conn(0x7f8598066a00 msgr2=0x7f8598066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:28.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.730+0000 7f859e8ff640 1 -- 172.21.15.145:0/3832646577 shutdown_connections 2024-03-29T11:42:28.736 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.730+0000 7f859e8ff640 1 -- 172.21.15.145:0/3832646577 wait complete. 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.731+0000 7f859e8ff640 10 monclient: build_initial_monmap 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.731+0000 7f859e8ff640 10 monclient: monmap: 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:42:28.739 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.731+0000 7f859e8ff640 1 librados: starting msgr at 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.731+0000 7f859e8ff640 1 librados: starting objecter 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.731+0000 7f859e8ff640 1 Processor -- start 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.732+0000 7f859e8ff640 1 -- start start 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.732+0000 7f859e8ff640 1 librados: setting wanted keys 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.732+0000 7f859e8ff640 1 librados: calling monclient init 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.732+0000 7f859e8ff640 10 monclient: init 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f859e8ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f859e8ff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f859e8ff640 10 monclient(hunting): picked mon.noname-c con 0x7f85980a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f859e8ff640 10 monclient(hunting): picked mon.noname-b con 0x7f859813f650 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f859e8ff640 10 monclient(hunting): picked mon.noname-a con 0x7f85981457f0 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f85977fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f859813f650 0x7f85981df4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:35836/0 (socket says 172.21.15.145:35836) 2024-03-29T11:42:28.740 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f85977fe640 1 -- 172.21.15.145:0/204658877 learned_addr learned my addr 172.21.15.145:0/204658877 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85981dfbf0 con 0x7f85981457f0 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980710f0 con 0x7f859813f650 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f8594ff9640 10 client.?.objecter ms_handle_connect 0x7f85980a9d70 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f8594ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f8594ff9640 10 client.?.objecter ms_handle_connect 0x7f859813f650 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.733+0000 7f8594ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85980713d0 con 0x7f85980a9d70 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f859e8ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f859e8ff640 10 monclient(hunting): authenticate will time out at 1338.041137s 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 client.?.objecter ms_handle_connect 0x7f85981457f0 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766283008 0 0) 0x7f85980710f0 con 0x7f859813f650 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:28.741 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8584004eb0 con 0x7f859813f650 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444189585 0 0) 0x7f85980713d0 con 0x7f85980a9d70 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85980710f0 con 0x7f85980a9d70 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869469282 0 0) 0x7f8584004eb0 con 0x7f859813f650 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.734+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85980713d0 con 0x7f859813f650 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f858c001a50 con 0x7f859813f650 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:28.742 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137380760 0 0) 0x7f85981dfbf0 con 0x7f85981457f0 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:28.743 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8584004eb0 con 0x7f85981457f0 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2126771153 0 0) 0x7f85980710f0 con 0x7f85980a9d70 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85981dfbf0 con 0x7f85980a9d70 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8588001af0 con 0x7f85980a9d70 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:28.744 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 661286779 0 0) 0x7f8584004eb0 con 0x7f85981457f0 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85980710f0 con 0x7f85981457f0 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8590001b70 con 0x7f85981457f0 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:28.745 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 49163746 0 0) 0x7f85980713d0 con 0x7f859813f650 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 >> v1:172.21.15.145:6790/0 conn(0x7f85980a9d70 legacy=0x7f85981dedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 >> v1:172.21.15.145:6789/0 conn(0x7f85981457f0 legacy=0x7f85981e7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 monclient: found mon.b 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f859813a1c0 con 0x7f859813f650 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient: _finish_auth 0 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f8594ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:28.746 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f859e8ff640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f859e8ff640 10 monclient: _renew_subs 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f859e8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85981dbdf0 con 0x7f859813f650 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f859e8ff640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f859e8ff640 10 monclient: _renew_subs 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f859e8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.735+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85981dbab0 con 0x7f859813f650 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.736+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f858c0035f0 con 0x7f859813f650 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.736+0000 7f8594ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.736+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f858c003900 con 0x7f859813f650 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.736+0000 7f8594ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.736+0000 7f8594ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.736+0000 7f8594ff9640 10 monclient: dump: 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:28.747 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.748 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:28.748 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:28.748 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:28.748 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:28.748 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:28.748 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:28.748 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:28.748 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.736+0000 7f85967fc640 4 set_mon_vals no callback set 2024-03-29T11:42:28.748 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.736+0000 7f859e8ff640 1 librados: init done 2024-03-29T11:42:28.749 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.738+0000 7f859e8ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:28.749 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.738+0000 7f859e8ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:28.749 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.738+0000 7f859e8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:28.749 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.738+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8598075f10 con 0x7f859813f650 2024-03-29T11:42:28.749 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.739+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f858c003a60 con 0x7f859813f650 2024-03-29T11:42:28.749 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.739+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f858c084a30 con 0x7f859813f650 2024-03-29T11:42:28.749 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.739+0000 7f8594ff9640 10 client.4495.objecter ms_dispatch 0x7f8598000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:28.750 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.739+0000 7f8594ff9640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:28.750 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.739+0000 7f8594ff9640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:28.750 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.739+0000 7f8594ff9640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:42:28.750 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.740+0000 7f8597fff640 10 monclient: get_auth_request con 0x7f858406e1b0 auth_method 0 2024-03-29T11:42:28.750 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.741+0000 7f8594ff9640 10 client.4495.objecter ms_handle_connect 0x7f858406e1b0 2024-03-29T11:42:28.750 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.746+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f858c050140 con 0x7f859813f650 2024-03-29T11:42:28.750 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.746+0000 7f8594ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:28.750 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:28.746+0000 7f8594ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:29.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.139+0000 7f859e8ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:29.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.139+0000 7f859e8ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:29.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.139+0000 7f859e8ff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:29.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.139+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85981dbf80 con 0x7f859813f650 2024-03-29T11:42:29.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.140+0000 7f8594ff9640 1 -- 172.21.15.145:0/204658877 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f858c052240 con 0x7f859813f650 2024-03-29T11:42:29.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.140+0000 7f8594ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:29.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.140+0000 7f8594ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:29.143 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:42:29.143 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.144+0000 7f859e8ff640 10 librados: watch_flush enter 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.144+0000 7f859e8ff640 10 librados: watch_flush exit 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.144+0000 7f859e8ff640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.144+0000 7f859e8ff640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.145+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 >> v1:172.21.15.145:6816/227372239 conn(0x7f858406e1b0 legacy=0x7f8584070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.145+0000 7f859e8ff640 10 monclient: shutdown 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.145+0000 7f859e8ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.145+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 >> v1:172.21.15.203:6789/0 conn(0x7f859813f650 legacy=0x7f85981df4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.145+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 shutdown_connections 2024-03-29T11:42:29.147 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.145+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 >> 172.21.15.145:0/204658877 conn(0x7f8598066a00 msgr2=0x7f85980b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:29.148 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.145+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 shutdown_connections 2024-03-29T11:42:29.148 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.145+0000 7f859e8ff640 1 -- 172.21.15.145:0/204658877 wait complete. 2024-03-29T11:42:29.148 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:29.145+0000 7f859e8ff640 1 librados: shutdown 2024-03-29T11:42:29.162 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:35.972 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.123+0000 7f44e239d640 10 monclient: get_monmap_and_config 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.123+0000 7f44e239d640 10 monclient: build_initial_monmap 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.123+0000 7f44e239d640 10 monclient: monmap: 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:42:36.124614+0000 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:42:36.124614+0000 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:36.127 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:42:36.128 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:42:36.128 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:36.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.124+0000 7f44e239d640 1 Processor -- start 2024-03-29T11:42:36.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.125+0000 7f44e239d640 1 -- start start 2024-03-29T11:42:36.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.125+0000 7f44e239d640 10 monclient: init 2024-03-29T11:42:36.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:36.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:36.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 10 monclient(hunting): picked mon.noname-a con 0x7f44dc0b01a0 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:36.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 10 monclient(hunting): picked mon.noname-c con 0x7f44dc0af010 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:36.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 10 monclient(hunting): picked mon.noname-b con 0x7f44dc0ac5a0 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:36.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44dc0ae8a0 con 0x7f44dc0b01a0 2024-03-29T11:42:36.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44dc0ab1d0 con 0x7f44dc0af010 2024-03-29T11:42:36.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44dc0ab4b0 con 0x7f44dc0ac5a0 2024-03-29T11:42:36.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:36.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.126+0000 7f44e239d640 10 monclient(hunting): authenticate will time out at 1345.434357s 2024-03-29T11:42:36.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.132+0000 7f44e093a640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f44dc0ac5a0 0x7f44dc151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:46545/0 (socket says 172.21.15.145:46545) 2024-03-29T11:42:36.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.132+0000 7f44e093a640 1 -- 172.21.15.145:0/1929930281 learned_addr learned my addr 172.21.15.145:0/1929930281 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106714929 0 0) 0x7f44dc0ab1d0 con 0x7f44dc0af010 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c8003520 con 0x7f44dc0af010 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108910943 0 0) 0x7f44dc0ae8a0 con 0x7f44dc0b01a0 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient(hunting): _negotiate 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:36.139 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c8003840 con 0x7f44dc0b01a0 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1440707360 0 0) 0x7f44c8003840 con 0x7f44dc0b01a0 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 >> v1:172.21.15.145:6790/0 conn(0x7f44dc0ac5a0 legacy=0x7f44dc151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 >> v1:172.21.15.203:6789/0 conn(0x7f44dc0af010 legacy=0x7f44dc149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient: _finish_hunting 0 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 1 monclient: found mon.noname-a 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc0ab790 con 0x7f44dc0b01a0 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient: _finish_auth 0 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 10 monclient: _check_auth_tickets 2024-03-29T11:42:36.140 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44e239d640 5 monclient: authenticate success, global_id 4497 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.134+0000 7f44e239d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.135+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f44cc0015f0 con 0x7f44dc0b01a0 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.135+0000 7f44daffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.135+0000 7f44daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.135+0000 7f44daffd640 10 monclient: dump: 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:36.141 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:36.142 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:36.142 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:36.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.135+0000 7f44e239d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:36.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.135+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44cc0017d0 con 0x7f44dc0b01a0 2024-03-29T11:42:36.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.135+0000 7f44daffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:36.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.135+0000 7f44e239d640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:36.142 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.135+0000 7f44e239d640 4 set_mon_vals no callback set 2024-03-29T11:42:36.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.136+0000 7f44e239d640 10 monclient: shutdown 2024-03-29T11:42:36.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.136+0000 7f44e239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:36.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.136+0000 7f44e239d640 1 -- 172.21.15.145:0/1929930281 >> v1:172.21.15.145:6789/0 conn(0x7f44dc0b01a0 legacy=0x7f44dc0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:36.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.136+0000 7f44daffd640 1 -- 172.21.15.145:0/1929930281 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f44cc0015f0 con 0x7f44dc0b01a0 2024-03-29T11:42:36.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.136+0000 7f44daffd640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:42:36.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.136+0000 7f44e239d640 1 -- 172.21.15.145:0/1929930281 shutdown_connections 2024-03-29T11:42:36.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.136+0000 7f44e239d640 1 -- 172.21.15.145:0/1929930281 >> 172.21.15.145:0/1929930281 conn(0x7f44dc06b660 msgr2=0x7f44dc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:36.143 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.137+0000 7f44e239d640 1 -- 172.21.15.145:0/1929930281 shutdown_connections 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.137+0000 7f44e239d640 1 -- 172.21.15.145:0/1929930281 wait complete. 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.138+0000 7f44e239d640 10 monclient: build_initial_monmap 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.138+0000 7f44e239d640 10 monclient: monmap: 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:36.144 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.138+0000 7f44e239d640 1 librados: starting msgr at 2024-03-29T11:42:36.145 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.138+0000 7f44e239d640 1 librados: starting objecter 2024-03-29T11:42:36.145 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.138+0000 7f44e239d640 1 Processor -- start 2024-03-29T11:42:36.145 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.141+0000 7f44e239d640 1 -- start start 2024-03-29T11:42:36.145 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.141+0000 7f44e239d640 1 librados: setting wanted keys 2024-03-29T11:42:36.145 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.141+0000 7f44e239d640 1 librados: calling monclient init 2024-03-29T11:42:36.146 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.141+0000 7f44e239d640 10 monclient: init 2024-03-29T11:42:36.148 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.142+0000 7f44e239d640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:36.149 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.142+0000 7f44e239d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:42:36.149 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.142+0000 7f44e239d640 10 monclient(hunting): picked mon.noname-b con 0x7f44dc0ac5a0 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:36.149 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.142+0000 7f44e239d640 10 monclient(hunting): picked mon.noname-c con 0x7f44dc0b01a0 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:36.149 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.145+0000 7f44dbfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f44dc0ac5a0 0x7f44dc1df280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:48997/0 (socket says 172.21.15.145:48997) 2024-03-29T11:42:36.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.148+0000 7f44e239d640 10 monclient(hunting): picked mon.noname-a con 0x7f44dc145120 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:36.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.148+0000 7f44e239d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44dc1df990 con 0x7f44dc145120 2024-03-29T11:42:36.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.148+0000 7f44e239d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44dc1dfc50 con 0x7f44dc0ac5a0 2024-03-29T11:42:36.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.148+0000 7f44e239d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44dc1286a0 con 0x7f44dc0b01a0 2024-03-29T11:42:36.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.148+0000 7f44e239d640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:36.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.148+0000 7f44e239d640 10 monclient(hunting): authenticate will time out at 1345.456646s 2024-03-29T11:42:36.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.148+0000 7f44e093a640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f44dc145120 0x7f44dc126a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:42563/0 (socket says 172.21.15.145:42563) 2024-03-29T11:42:36.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.148+0000 7f44e093a640 1 -- 172.21.15.145:0/2647638584 learned_addr learned my addr 172.21.15.145:0/2647638584 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:36.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 client.?.objecter ms_handle_connect 0x7f44dc145120 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 client.?.objecter ms_handle_connect 0x7f44dc0ac5a0 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2046173556 0 0) 0x7f44dc1df990 con 0x7f44dc145120 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44bc004eb0 con 0x7f44dc145120 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476463852 0 0) 0x7f44dc1dfc50 con 0x7f44dc0ac5a0 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.149+0000 7f44d8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.150+0000 7f44d8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:36.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.150+0000 7f44d8ff9640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.150+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44dc1df990 con 0x7f44dc0ac5a0 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.150+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 900467441 0 0) 0x7f44bc004eb0 con 0x7f44dc145120 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.150+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44dc1dfc50 con 0x7f44dc145120 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.150+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3901131756 0 0) 0x7f44dc1df990 con 0x7f44dc0ac5a0 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.150+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44bc004eb0 con 0x7f44dc0ac5a0 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f44c40018b0 con 0x7f44dc145120 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:36.157 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f44cc001b60 con 0x7f44dc0ac5a0 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 10 monclient(hunting): dump: 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:36.158 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:36.159 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:36.159 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:36.159 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:36.159 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:36.159 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:36.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1929948672 0 0) 0x7f44dc1dfc50 con 0x7f44dc145120 2024-03-29T11:42:36.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.151+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 >> v1:172.21.15.145:6790/0 conn(0x7f44dc0b01a0 legacy=0x7f44dc126320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.152+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 >> v1:172.21.15.203:6789/0 conn(0x7f44dc0ac5a0 legacy=0x7f44dc1df280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.152+0000 7f44d8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.152+0000 7f44d8ff9640 1 monclient: found mon.a 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.152+0000 7f44d8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44d8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc1ea590 con 0x7f44dc145120 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44d8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44d8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44e239d640 5 monclient: authenticate success, global_id 4500 2024-03-29T11:42:36.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44e239d640 10 monclient: _renew_subs 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44e239d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44dc1e7560 con 0x7f44dc145120 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44e239d640 10 client.4500.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44e239d640 10 monclient: _renew_subs 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44e239d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44dc1e7a30 con 0x7f44dc145120 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44e239d640 1 librados: init done 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44c40047e0 con 0x7f44dc145120 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.153+0000 7f44d8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.154+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f44c4004bf0 con 0x7f44dc145120 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.154+0000 7f44d8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.154+0000 7f44d8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.154+0000 7f44d8ff9640 10 monclient: dump: 2024-03-29T11:42:36.163 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.154+0000 7f44d9ffb640 4 set_mon_vals no callback set 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.155+0000 7f44e239d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.155+0000 7f44e239d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.155+0000 7f44e239d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:36.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.155+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c0005080 con 0x7f44dc145120 2024-03-29T11:42:36.165 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.159+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f44c401abc0 con 0x7f44dc145120 2024-03-29T11:42:36.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.160+0000 7f44dbfff640 10 monclient: get_auth_request con 0x7f44bc06dee0 auth_method 0 2024-03-29T11:42:36.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.161+0000 7f44d8ff9640 10 client.4500.objecter ms_handle_connect 0x7f44bc06dee0 2024-03-29T11:42:36.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.161+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f44c4084a00 con 0x7f44dc145120 2024-03-29T11:42:36.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.161+0000 7f44d8ff9640 10 client.4500.objecter ms_dispatch 0x7f44dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:36.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.161+0000 7f44d8ff9640 3 client.4500.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:36.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.161+0000 7f44d8ff9640 3 client.4500.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:36.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.161+0000 7f44d8ff9640 20 client.4500.objecter dump_active .. 0 homeless 2024-03-29T11:42:36.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.169+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44c40502e0 con 0x7f44dc145120 2024-03-29T11:42:36.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.169+0000 7f44d8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:36.177 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.169+0000 7f44d8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:36.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.545+0000 7f44e239d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:36.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.545+0000 7f44e239d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:36.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.545+0000 7f44e239d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:36.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.545+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44c0005330 con 0x7f44dc145120 2024-03-29T11:42:36.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.546+0000 7f44d8ff9640 1 -- 172.21.15.145:0/2647638584 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f44c4052210 con 0x7f44dc145120 2024-03-29T11:42:36.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.546+0000 7f44d8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:36.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.546+0000 7f44d8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:36.550 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:42:36.550 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:36.554 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.550+0000 7f44e239d640 10 librados: watch_flush enter 2024-03-29T11:42:36.554 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.551+0000 7f44e239d640 10 librados: watch_flush exit 2024-03-29T11:42:36.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.551+0000 7f44e239d640 20 client.4500.objecter shutdown clearing up homeless session... 2024-03-29T11:42:36.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.551+0000 7f44e239d640 10 client.4500.objecter successfully canceled tick 2024-03-29T11:42:36.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.551+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 >> v1:172.21.15.145:6816/227372239 conn(0x7f44bc06dee0 legacy=0x7f44bc0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:36.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.551+0000 7f44e239d640 10 monclient: shutdown 2024-03-29T11:42:36.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.551+0000 7f44e239d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:36.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.551+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 >> v1:172.21.15.145:6789/0 conn(0x7f44dc145120 legacy=0x7f44dc126a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:36.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.551+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 shutdown_connections 2024-03-29T11:42:36.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.551+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 >> 172.21.15.145:0/2647638584 conn(0x7f44dc06b660 msgr2=0x7f44dc149480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:36.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.552+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 shutdown_connections 2024-03-29T11:42:36.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.552+0000 7f44e239d640 1 -- 172.21.15.145:0/2647638584 wait complete. 2024-03-29T11:42:36.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:36.552+0000 7f44e239d640 1 librados: shutdown 2024-03-29T11:42:36.572 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:43.379 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.526+0000 7f2bfde27640 10 monclient: get_monmap_and_config 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.526+0000 7f2bfde27640 10 monclient: build_initial_monmap 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.526+0000 7f2bfde27640 10 monclient: monmap: 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:42:43.527907+0000 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:42:43.527907+0000 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:42:43.530 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:42:43.531 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:43.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.528+0000 7f2bfde27640 1 Processor -- start 2024-03-29T11:42:43.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.528+0000 7f2bfde27640 1 -- start start 2024-03-29T11:42:43.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.528+0000 7f2bfde27640 10 monclient: init 2024-03-29T11:42:43.535 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.528+0000 7f2bfde27640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:43.536 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.528+0000 7f2bfde27640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:42:43.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.529+0000 7f2bfde27640 10 monclient(hunting): picked mon.noname-c con 0x7f2bf80b1f60 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:43.537 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.529+0000 7f2bfde27640 10 monclient(hunting): picked mon.noname-a con 0x7f2bf8071eb0 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:43.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.529+0000 7f2bfde27640 10 monclient(hunting): picked mon.noname-b con 0x7f2bf8143d10 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:43.538 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.529+0000 7f2bfde27640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf806ef80 con 0x7f2bf8071eb0 2024-03-29T11:42:43.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.529+0000 7f2bfde27640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf806f260 con 0x7f2bf80b1f60 2024-03-29T11:42:43.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.529+0000 7f2bfde27640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80b40c0 con 0x7f2bf8143d10 2024-03-29T11:42:43.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.529+0000 7f2bfde27640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:43.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.529+0000 7f2bfde27640 10 monclient(hunting): authenticate will time out at 1352.837109s 2024-03-29T11:42:43.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.533+0000 7f2bf7fff640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f2bf8143d10 0x7f2bf8146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:38261/0 (socket says 172.21.15.145:38261) 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.533+0000 7f2bf7fff640 1 -- 172.21.15.145:0/1221385575 learned_addr learned my addr 172.21.15.145:0/1221385575 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.534+0000 7f2bf67fc640 1 -- 172.21.15.145:0/1221385575 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2206091435 0 0) 0x7f2bf80b40c0 con 0x7f2bf8143d10 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.534+0000 7f2bf67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.534+0000 7f2bf67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.534+0000 7f2bf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.534+0000 7f2bf67fc640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.534+0000 7f2bf67fc640 1 -- 172.21.15.145:0/1221385575 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be0003520 con 0x7f2bf8143d10 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.534+0000 7f2bf67fc640 1 -- 172.21.15.145:0/1221385575 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059894998 0 0) 0x7f2be0003520 con 0x7f2bf8143d10 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.534+0000 7f2bf67fc640 1 -- 172.21.15.145:0/1221385575 >> v1:172.21.15.203:6789/0 conn(0x7f2bf80b1f60 legacy=0x7f2bf8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.534+0000 7f2bf67fc640 1 -- 172.21.15.145:0/1221385575 >> v1:172.21.15.145:6789/0 conn(0x7f2bf8071eb0 legacy=0x7f2bf8146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 1 monclient: found mon.noname-b 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:42:43.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 1 -- 172.21.15.145:0/1221385575 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf8155d70 con 0x7f2bf8143d10 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bfde27640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bfde27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 1 -- 172.21.15.145:0/1221385575 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2bf0001a10 con 0x7f2bf8143d10 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bf67fc640 10 monclient: dump: 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:43.544 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.536+0000 7f2bfde27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.537+0000 7f2bf67fc640 1 -- 172.21.15.145:0/1221385575 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bf0001ca0 con 0x7f2bf8143d10 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.537+0000 7f2bf67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.537+0000 7f2bf67fc640 1 -- 172.21.15.145:0/1221385575 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2bf0002030 con 0x7f2bf8143d10 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.537+0000 7f2bf67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.537+0000 7f2bf67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.537+0000 7f2bf67fc640 10 monclient: dump: 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:43.545 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.537+0000 7f2bfde27640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.537+0000 7f2bfde27640 4 set_mon_vals no callback set 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.538+0000 7f2bfde27640 10 monclient: shutdown 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.538+0000 7f2bfde27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:43.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.538+0000 7f2bfde27640 1 -- 172.21.15.145:0/1221385575 >> v1:172.21.15.145:6790/0 conn(0x7f2bf8143d10 legacy=0x7f2bf8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.547 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.540+0000 7f2bfde27640 1 -- 172.21.15.145:0/1221385575 shutdown_connections 2024-03-29T11:42:43.547 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.540+0000 7f2bfde27640 1 -- 172.21.15.145:0/1221385575 >> 172.21.15.145:0/1221385575 conn(0x7f2bf8066a00 msgr2=0x7f2bf8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:43.547 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.542+0000 7f2bfde27640 1 -- 172.21.15.145:0/1221385575 shutdown_connections 2024-03-29T11:42:43.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.542+0000 7f2bfde27640 1 -- 172.21.15.145:0/1221385575 wait complete. 2024-03-29T11:42:43.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.543+0000 7f2bfde27640 10 monclient: build_initial_monmap 2024-03-29T11:42:43.548 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.543+0000 7f2bfde27640 10 monclient: monmap: 2024-03-29T11:42:43.548 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:43.548 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.543+0000 7f2bfde27640 1 librados: starting msgr at 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.543+0000 7f2bfde27640 1 librados: starting objecter 2024-03-29T11:42:43.549 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.544+0000 7f2bfde27640 1 Processor -- start 2024-03-29T11:42:43.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.544+0000 7f2bfde27640 1 -- start start 2024-03-29T11:42:43.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.544+0000 7f2bfde27640 1 librados: setting wanted keys 2024-03-29T11:42:43.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.544+0000 7f2bfde27640 1 librados: calling monclient init 2024-03-29T11:42:43.550 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.544+0000 7f2bfde27640 10 monclient: init 2024-03-29T11:42:43.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:43.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:42:43.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 10 monclient(hunting): picked mon.noname-a con 0x7f2bf80b1f60 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:43.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 10 monclient(hunting): picked mon.noname-c con 0x7f2bf8143d10 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:43.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 10 monclient(hunting): picked mon.noname-b con 0x7f2bf8145620 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:43.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf81e42a0 con 0x7f2bf80b1f60 2024-03-29T11:42:43.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80ac7f0 con 0x7f2bf8145620 2024-03-29T11:42:43.551 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf80ab640 con 0x7f2bf8143d10 2024-03-29T11:42:43.552 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:43.552 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.545+0000 7f2bfde27640 10 monclient(hunting): authenticate will time out at 1352.853152s 2024-03-29T11:42:43.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.550+0000 7f2bf7fff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f2bf8145620 0x7f2bf81eeb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:57831/0 (socket says 172.21.15.145:57831) 2024-03-29T11:42:43.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.550+0000 7f2bf7fff640 1 -- 172.21.15.145:0/1263066328 learned_addr learned my addr 172.21.15.145:0/1263066328 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:43.555 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.550+0000 7f2bdffff640 10 client.?.objecter ms_handle_connect 0x7f2bf8145620 2024-03-29T11:42:43.556 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.551+0000 7f2bdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:43.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.553+0000 7f2bdffff640 10 client.?.objecter ms_handle_connect 0x7f2bf80b1f60 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.553+0000 7f2bdffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.553+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754875986 0 0) 0x7f2bf81e42a0 con 0x7f2bf80b1f60 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.553+0000 7f2bdffff640 10 monclient(hunting): _negotiate 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.553+0000 7f2bdffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.553+0000 7f2bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.553+0000 7f2bdffff640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.553+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2be4004eb0 con 0x7f2bf80b1f60 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.553+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1219076915 0 0) 0x7f2be4004eb0 con 0x7f2bf80b1f60 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2be4005300 con 0x7f2bf80b1f60 2024-03-29T11:42:43.562 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2bec001930 con 0x7f2bf80b1f60 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 10 monclient(hunting): dump: 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:43.563 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3338514458 0 0) 0x7f2be4005300 con 0x7f2bf80b1f60 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 >> v1:172.21.15.145:6790/0 conn(0x7f2bf8143d10 legacy=0x7f2bf81e4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 >> v1:172.21.15.203:6789/0 conn(0x7f2bf8145620 legacy=0x7f2bf81eeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 10 monclient: _finish_hunting 0 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 1 monclient: found mon.a 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf81e4e90 con 0x7f2bf80b1f60 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 10 monclient: _finish_auth 0 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 10 monclient: _check_auth_tickets 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bec003610 con 0x7f2bf80b1f60 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.554+0000 7f2bf5ffb640 4 set_mon_vals no callback set 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.555+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f2bec0039a0 con 0x7f2bf80b1f60 2024-03-29T11:42:43.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.555+0000 7f2bdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.555+0000 7f2bdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.555+0000 7f2bdffff640 10 monclient: dump: 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:43.565 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:43.567 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.558+0000 7f2bfde27640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:42:43.567 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.558+0000 7f2bfde27640 10 monclient: _renew_subs 2024-03-29T11:42:43.567 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.558+0000 7f2bfde27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:43.567 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.558+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf80aa7c0 con 0x7f2bf80b1f60 2024-03-29T11:42:43.567 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.559+0000 7f2bfde27640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:43.567 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.559+0000 7f2bfde27640 10 monclient: _renew_subs 2024-03-29T11:42:43.567 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.559+0000 7f2bfde27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:43.567 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.559+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf81f0440 con 0x7f2bf80b1f60 2024-03-29T11:42:43.567 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.559+0000 7f2bfde27640 1 librados: init done 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.560+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f2bec019970 con 0x7f2bf80b1f60 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.561+0000 7f2bfde27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.561+0000 7f2bfde27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.561+0000 7f2bfde27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.561+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf81e4100 con 0x7f2bf80b1f60 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.561+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f2bec083670 con 0x7f2bf80b1f60 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.561+0000 7f2bdffff640 10 client.4503.objecter ms_dispatch 0x7f2bf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.561+0000 7f2bdffff640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.561+0000 7f2bdffff640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.561+0000 7f2bdffff640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.562+0000 7f2bf6ffd640 10 monclient: get_auth_request con 0x7f2be406e150 auth_method 0 2024-03-29T11:42:43.568 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.562+0000 7f2bdffff640 10 client.4503.objecter ms_handle_connect 0x7f2be406e150 2024-03-29T11:42:43.573 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.569+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bf81e4100 con 0x7f2bf80b1f60 2024-03-29T11:42:43.573 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.569+0000 7f2bdffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:43.573 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.569+0000 7f2bdffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:43.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.890+0000 7f2bfde27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:43.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.890+0000 7f2bfde27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:43.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.890+0000 7f2bfde27640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:42:43.893 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.890+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bf80aa950 con 0x7f2bf80b1f60 2024-03-29T11:42:43.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.890+0000 7f2bdffff640 1 -- 172.21.15.145:0/1263066328 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f2bf81e4100 con 0x7f2bf80b1f60 2024-03-29T11:42:43.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.892+0000 7f2bdffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:43.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.892+0000 7f2bdffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:43.897 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:42:43.897 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:43.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.896+0000 7f2bfde27640 10 librados: watch_flush enter 2024-03-29T11:42:43.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.896+0000 7f2bfde27640 10 librados: watch_flush exit 2024-03-29T11:42:43.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.896+0000 7f2bfde27640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-29T11:42:43.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.896+0000 7f2bfde27640 10 client.4503.objecter successfully canceled tick 2024-03-29T11:42:43.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.897+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 >> v1:172.21.15.145:6816/227372239 conn(0x7f2be406e150 legacy=0x7f2be4070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.897+0000 7f2bfde27640 10 monclient: shutdown 2024-03-29T11:42:43.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.897+0000 7f2bfde27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:43.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.897+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 >> v1:172.21.15.145:6789/0 conn(0x7f2bf80b1f60 legacy=0x7f2bf80aaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:43.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.898+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 shutdown_connections 2024-03-29T11:42:43.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.898+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 >> 172.21.15.145:0/1263066328 conn(0x7f2bf8066a00 msgr2=0x7f2bf80b6e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:43.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.899+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 shutdown_connections 2024-03-29T11:42:43.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.899+0000 7f2bfde27640 1 -- 172.21.15.145:0/1263066328 wait complete. 2024-03-29T11:42:43.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:43.899+0000 7f2bfde27640 1 librados: shutdown 2024-03-29T11:42:43.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:50.729 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:50.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.871+0000 7f0c1ef66640 10 monclient: get_monmap_and_config 2024-03-29T11:42:50.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.871+0000 7f0c1ef66640 10 monclient: build_initial_monmap 2024-03-29T11:42:50.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.871+0000 7f0c1ef66640 10 monclient: monmap: 2024-03-29T11:42:50.875 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:50.875 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:50.875 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:42:50.873193+0000 2024-03-29T11:42:50.875 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:42:50.873193+0000 2024-03-29T11:42:50.876 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:50.876 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:50.876 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:50.876 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:42:50.876 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:42:50.876 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:50.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.872+0000 7f0c1ef66640 1 Processor -- start 2024-03-29T11:42:50.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.873+0000 7f0c1ef66640 1 -- start start 2024-03-29T11:42:50.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.873+0000 7f0c1ef66640 10 monclient: init 2024-03-29T11:42:50.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1ef66640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:50.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1ef66640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:42:50.881 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1ef66640 10 monclient(hunting): picked mon.noname-a con 0x7f0c100389e0 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1ef66640 10 monclient(hunting): picked mon.noname-c con 0x7f0c10038dc0 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1d763640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f0c10038dc0 0x7f0c100cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:49482/0 (socket says 172.21.15.145:49482) 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1e765640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f0c10030660 0x7f0c100d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:49874/0 (socket says 172.21.15.145:49874) 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1ef66640 10 monclient(hunting): picked mon.noname-b con 0x7f0c10030660 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1d763640 1 -- 172.21.15.145:0/3765295858 learned_addr learned my addr 172.21.15.145:0/3765295858 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1ef66640 1 -- 172.21.15.145:0/3765295858 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c100c54c0 con 0x7f0c100389e0 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.874+0000 7f0c1ef66640 1 -- 172.21.15.145:0/3765295858 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c10036430 con 0x7f0c10038dc0 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1ef66640 1 -- 172.21.15.145:0/3765295858 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c10039840 con 0x7f0c10030660 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1ef66640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1ef66640 10 monclient(hunting): authenticate will time out at 1360.183051s 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878149502 0 0) 0x7f0c10039840 con 0x7f0c10030660 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c04003520 con 0x7f0c10030660 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27896736 0 0) 0x7f0c10036430 con 0x7f0c10038dc0 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c10039840 con 0x7f0c10038dc0 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482987104 0 0) 0x7f0c100c54c0 con 0x7f0c100389e0 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:42:50.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.875+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c10036430 con 0x7f0c100389e0 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 461224277 0 0) 0x7f0c04003520 con 0x7f0c10030660 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 >> v1:172.21.15.203:6789/0 conn(0x7f0c10038dc0 legacy=0x7f0c100cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 >> v1:172.21.15.145:6789/0 conn(0x7f0c100389e0 legacy=0x7f0c10033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: _finish_hunting 0 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 1 monclient: found mon.noname-b 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c10035690 con 0x7f0c10030660 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: _finish_auth 0 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: _check_auth_tickets 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0c1804f730 con 0x7f0c10030660 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:50.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: dump: 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1ef66640 5 monclient: authenticate success, global_id 4535 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1ef66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c180008d0 con 0x7f0c10030660 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 1 -- 172.21.15.145:0/3765295858 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0c180758b0 con 0x7f0c10030660 2024-03-29T11:42:50.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1cf62640 10 monclient: dump: 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1ef66640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.876+0000 7f0c1ef66640 4 set_mon_vals no callback set 2024-03-29T11:42:50.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.877+0000 7f0c1ef66640 10 monclient: shutdown 2024-03-29T11:42:50.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.877+0000 7f0c1ef66640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:50.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.877+0000 7f0c1ef66640 1 -- 172.21.15.145:0/3765295858 >> v1:172.21.15.145:6790/0 conn(0x7f0c10030660 legacy=0x7f0c100d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.878+0000 7f0c1ef66640 1 -- 172.21.15.145:0/3765295858 shutdown_connections 2024-03-29T11:42:50.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.878+0000 7f0c1ef66640 1 -- 172.21.15.145:0/3765295858 >> 172.21.15.145:0/3765295858 conn(0x7f0c10005c00 msgr2=0x7f0c10006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:50.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.878+0000 7f0c1ef66640 1 -- 172.21.15.145:0/3765295858 shutdown_connections 2024-03-29T11:42:50.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.878+0000 7f0c1ef66640 1 -- 172.21.15.145:0/3765295858 wait complete. 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.879+0000 7f0c1ef66640 10 monclient: build_initial_monmap 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.879+0000 7f0c1ef66640 10 monclient: monmap: 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:50.895 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.879+0000 7f0c1ef66640 1 librados: starting msgr at 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.879+0000 7f0c1ef66640 1 librados: starting objecter 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.880+0000 7f0c1ef66640 1 Processor -- start 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.880+0000 7f0c1ef66640 1 -- start start 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.880+0000 7f0c1ef66640 1 librados: setting wanted keys 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.880+0000 7f0c1ef66640 1 librados: calling monclient init 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.880+0000 7f0c1ef66640 10 monclient: init 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.880+0000 7f0c1ef66640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1ef66640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1ef66640 10 monclient(hunting): picked mon.noname-c con 0x7f0c10030660 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:50.896 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1ef66640 10 monclient(hunting): picked mon.noname-a con 0x7f0c100389e0 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:50.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1d763640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f0c100389e0 0x7f0c100c15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:40776/0 (socket says 172.21.15.145:40776) 2024-03-29T11:42:50.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1d763640 1 -- 172.21.15.145:0/1453627629 learned_addr learned my addr 172.21.15.145:0/1453627629 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:50.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1ef66640 10 monclient(hunting): picked mon.noname-b con 0x7f0c100c5650 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:50.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 client.?.objecter ms_handle_connect 0x7f0c10030660 2024-03-29T11:42:50.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:50.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 client.?.objecter ms_handle_connect 0x7f0c100389e0 2024-03-29T11:42:50.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c100c64f0 con 0x7f0c100389e0 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c100c6210 con 0x7f0c100c5650 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c100c2400 con 0x7f0c10030660 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1ef66640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c1ef66640 10 monclient(hunting): authenticate will time out at 1360.189733s 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 client.?.objecter ms_handle_connect 0x7f0c100c5650 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161583586 0 0) 0x7f0c100c2400 con 0x7f0c10030660 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bfc004eb0 con 0x7f0c10030660 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.881+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342992806 0 0) 0x7f0c100c64f0 con 0x7f0c100389e0 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c100c2400 con 0x7f0c100389e0 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529829169 0 0) 0x7f0c100c6210 con 0x7f0c100c5650 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c100c64f0 con 0x7f0c100c5650 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2003399580 0 0) 0x7f0c100c64f0 con 0x7f0c100c5650 2024-03-29T11:42:50.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c100c6210 con 0x7f0c100c5650 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1933365838 0 0) 0x7f0c100c2400 con 0x7f0c100389e0 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c100c64f0 con 0x7f0c100389e0 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0c1804f400 con 0x7f0c100c5650 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.882+0000 7f0c0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:50.900 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2686317224 0 0) 0x7f0bfc004eb0 con 0x7f0c10030660 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c100c2400 con 0x7f0c10030660 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0c080018f0 con 0x7f0c100389e0 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:50.901 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0c140018b0 con 0x7f0c10030660 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:50.902 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3213329352 0 0) 0x7f0c100c6210 con 0x7f0c100c5650 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 >> v1:172.21.15.145:6790/0 conn(0x7f0c10030660 legacy=0x7f0c10160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 >> v1:172.21.15.145:6789/0 conn(0x7f0c100389e0 legacy=0x7f0c100c15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 1 monclient: found mon.b 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:50.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c101616c0 con 0x7f0c100c5650 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c1ef66640 5 monclient: authenticate success, global_id 4513 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c1ef66640 10 monclient: _renew_subs 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c1ef66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.883+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c100c6980 con 0x7f0c100c5650 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c1804f730 con 0x7f0c100c5650 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0c18067d90 con 0x7f0c100c5650 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c0e7fc640 10 monclient: dump: 2024-03-29T11:42:50.904 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:50.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c1ef66640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c1ef66640 10 monclient: _renew_subs 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c1ef66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c1000ce20 con 0x7f0c100c5650 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c0f7fe640 4 set_mon_vals no callback set 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.884+0000 7f0c1ef66640 1 librados: init done 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.885+0000 7f0c1ef66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.885+0000 7f0c1ef66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.885+0000 7f0c1ef66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.885+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c100c66a0 con 0x7f0c100c5650 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.887+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f0c18067ef0 con 0x7f0c100c5650 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.888+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f0c180ff330 con 0x7f0c100c5650 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.888+0000 7f0c0e7fc640 10 client.4513.objecter ms_dispatch 0x7f0c18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.888+0000 7f0c0e7fc640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.888+0000 7f0c0e7fc640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:50.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.888+0000 7f0c0e7fc640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-29T11:42:50.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.888+0000 7f0c1df64640 10 monclient: get_auth_request con 0x7f0bfc06e020 auth_method 0 2024-03-29T11:42:50.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.888+0000 7f0c0e7fc640 10 client.4513.objecter ms_handle_connect 0x7f0bfc06e020 2024-03-29T11:42:50.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.895+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c100c66a0 con 0x7f0c100c5650 2024-03-29T11:42:50.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.895+0000 7f0c0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:50.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:50.895+0000 7f0c0e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:51.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.273+0000 7f0c1ef66640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:51.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.273+0000 7f0c1ef66640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:51.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.273+0000 7f0c1ef66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:42:51.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.273+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c100c66a0 con 0x7f0c100c5650 2024-03-29T11:42:51.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.274+0000 7f0c0e7fc640 1 -- 172.21.15.145:0/1453627629 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f0c100c66a0 con 0x7f0c100c5650 2024-03-29T11:42:51.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.274+0000 7f0c0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:51.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.274+0000 7f0c0e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:51.278 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:42:51.279 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:51.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 10 librados: watch_flush enter 2024-03-29T11:42:51.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 10 librados: watch_flush exit 2024-03-29T11:42:51.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-29T11:42:51.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 10 client.4513.objecter successfully canceled tick 2024-03-29T11:42:51.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 >> v1:172.21.15.145:6816/227372239 conn(0x7f0bfc06e020 legacy=0x7f0bfc0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:51.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 10 monclient: shutdown 2024-03-29T11:42:51.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:51.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 >> v1:172.21.15.203:6789/0 conn(0x7f0c100c5650 legacy=0x7f0c100c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:51.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 shutdown_connections 2024-03-29T11:42:51.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 >> 172.21.15.145:0/1453627629 conn(0x7f0c10005c00 msgr2=0x7f0c10034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:51.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.278+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 shutdown_connections 2024-03-29T11:42:51.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.279+0000 7f0c1ef66640 1 -- 172.21.15.145:0/1453627629 wait complete. 2024-03-29T11:42:51.285 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:51.279+0000 7f0c1ef66640 1 librados: shutdown 2024-03-29T11:42:51.297 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:42:58.104 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.254+0000 7f4f5ffd6640 10 monclient: get_monmap_and_config 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.254+0000 7f4f5ffd6640 10 monclient: build_initial_monmap 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.254+0000 7f4f5ffd6640 10 monclient: monmap: 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:42:58.255817+0000 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:42:58.255817+0000 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:58.258 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:42:58.259 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:42:58.259 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:58.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.255+0000 7f4f5ffd6640 1 Processor -- start 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.255+0000 7f4f5ffd6640 1 -- start start 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.255+0000 7f4f5ffd6640 10 monclient: init 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f580b1f60 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f58071eb0 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f58143d10 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5806ef80 con 0x7f4f58143d10 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5806f260 con 0x7f4f58071eb0 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f580b40c0 con 0x7f4f580b1f60 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:58.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.256+0000 7f4f5ffd6640 10 monclient(hunting): authenticate will time out at 1367.564760s 2024-03-29T11:42:58.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.259+0000 7f4f5dd72640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f4f580b1f60 0x7f4f58143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:55711/0 (socket says 172.21.15.145:55711) 2024-03-29T11:42:58.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.259+0000 7f4f5dd72640 1 -- 172.21.15.145:0/1806354803 learned_addr learned my addr 172.21.15.145:0/1806354803 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:58.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.260+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018907624 0 0) 0x7f4f580b40c0 con 0x7f4f580b1f60 2024-03-29T11:42:58.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.260+0000 7f4f5cd70640 10 monclient(hunting): _negotiate 2024-03-29T11:42:58.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.260+0000 7f4f5cd70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:58.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.260+0000 7f4f5cd70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:58.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.260+0000 7f4f5cd70640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:42:58.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.260+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f44003520 con 0x7f4f580b1f60 2024-03-29T11:42:58.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.261+0000 7f4f5dd72640 1 -- 172.21.15.145:0/1806354803 >> v1:172.21.15.145:6790/0 conn(0x7f4f580b1f60 legacy=0x7f4f58143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:42:58.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.261+0000 7f4f5dd72640 1 -- 172.21.15.145:0/1806354803 >> v1:172.21.15.145:6790/0 conn(0x7f4f580b1f60 legacy=0x7f4f58143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:42:58.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.261+0000 7f4f5dd72640 1 --1- 172.21.15.145:0/1806354803 >> v1:172.21.15.145:6790/0 conn(0x7f4f580b1f60 0x7f4f58143600 :-1 s=OPENED pgs=745010 cs=1 l=1).handle_message read tag failed 2024-03-29T11:42:58.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.261+0000 7f4f5dd72640 1 --1- 172.21.15.145:0/1806354803 >> v1:172.21.15.145:6790/0 conn(0x7f4f580b1f60 0x7f4f58143600 :-1 s=OPENED pgs=745010 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:42:58.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.261+0000 7f4f5cd70640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.145:6790/0 2024-03-29T11:42:58.267 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751278508 0 0) 0x7f4f5806f260 con 0x7f4f58071eb0 2024-03-29T11:42:58.267 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 10 monclient(hunting): _negotiate 2024-03-29T11:42:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:42:58.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f44003840 con 0x7f4f58071eb0 2024-03-29T11:42:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112493941 0 0) 0x7f4f5806ef80 con 0x7f4f58143d10 2024-03-29T11:42:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 10 monclient(hunting): _negotiate 2024-03-29T11:42:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:42:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.264+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f44003b80 con 0x7f4f58143d10 2024-03-29T11:42:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1703196425 0 0) 0x7f4f44003b80 con 0x7f4f58143d10 2024-03-29T11:42:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 >> v1:172.21.15.145:6790/0 conn(0x7f4f580b1f60 legacy=0x7f4f58143600 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:42:58.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 >> v1:172.21.15.203:6789/0 conn(0x7f4f58071eb0 legacy=0x7f4f58146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: _finish_hunting 0 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 1 monclient: found mon.noname-a 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f58155d70 con 0x7f4f58143d10 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: _finish_auth 0 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: _check_auth_tickets 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4f500018f0 con 0x7f4f58143d10 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: dump: 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:58.273 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5ffd6640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5ffd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f50001a50 con 0x7f4f58143d10 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5ffd6640 10 monclient: get_monmap_and_config success 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5ffd6640 4 set_mon_vals no callback set 2024-03-29T11:42:58.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 1 -- 172.21.15.145:0/1806354803 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4f50001ea0 con 0x7f4f58143d10 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.265+0000 7f4f5cd70640 10 monclient: dump: 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.266+0000 7f4f5ffd6640 10 monclient: shutdown 2024-03-29T11:42:58.275 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.266+0000 7f4f5ffd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.266+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/1806354803 >> v1:172.21.15.145:6789/0 conn(0x7f4f58143d10 legacy=0x7f4f58146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.266+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/1806354803 shutdown_connections 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.266+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/1806354803 >> 172.21.15.145:0/1806354803 conn(0x7f4f58066a00 msgr2=0x7f4f58066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.267+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/1806354803 shutdown_connections 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.267+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/1806354803 wait complete. 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.268+0000 7f4f5ffd6640 10 monclient: build_initial_monmap 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.268+0000 7f4f5ffd6640 10 monclient: monmap: 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:42:58.276 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:42:58.277 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:58.282 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.268+0000 7f4f5ffd6640 1 librados: starting msgr at 2024-03-29T11:42:58.282 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.268+0000 7f4f5ffd6640 1 librados: starting objecter 2024-03-29T11:42:58.282 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.268+0000 7f4f5ffd6640 1 Processor -- start 2024-03-29T11:42:58.282 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.269+0000 7f4f5ffd6640 1 -- start start 2024-03-29T11:42:58.282 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.269+0000 7f4f5ffd6640 1 librados: setting wanted keys 2024-03-29T11:42:58.282 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.269+0000 7f4f5ffd6640 1 librados: calling monclient init 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.269+0000 7f4f5ffd6640 10 monclient: init 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.269+0000 7f4f5ffd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.269+0000 7f4f5ffd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.269+0000 7f4f5ffd6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f580b1f60 addr v1:172.21.15.145:6790/0 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.272+0000 7f4f5ffd6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f58143d10 addr v1:172.21.15.203:6789/0 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.274+0000 7f4f5dd72640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f4f580b1f60 0x7f4f5813c500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:55815/0 (socket says 172.21.15.145:55815) 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.274+0000 7f4f5dd72640 1 -- 172.21.15.145:0/2975830038 learned_addr learned my addr 172.21.15.145:0/2975830038 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.274+0000 7f4f5ffd6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f58139480 addr v1:172.21.15.145:6789/0 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.274+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5813cc10 con 0x7f4f58139480 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.274+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5813d960 con 0x7f4f58143d10 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.274+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f5813dc40 con 0x7f4f580b1f60 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.274+0000 7f4f5ffd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.274+0000 7f4f5ffd6640 10 monclient(hunting): authenticate will time out at 1367.582835s 2024-03-29T11:42:58.283 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 10 client.?.objecter ms_handle_connect 0x7f4f580b1f60 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651783868 0 0) 0x7f4f5813dc40 con 0x7f4f580b1f60 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f3c004eb0 con 0x7f4f580b1f60 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 10 client.?.objecter ms_handle_connect 0x7f4f58143d10 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.275+0000 7f4f4e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2866292630 0 0) 0x7f4f3c004eb0 con 0x7f4f580b1f60 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f5813dc40 con 0x7f4f580b1f60 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4f54001a10 con 0x7f4f580b1f60 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:42:58.284 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient(hunting): dump: 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898482763 0 0) 0x7f4f5813d960 con 0x7f4f58143d10 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:42:58.285 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f3c004eb0 con 0x7f4f58143d10 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 56401543 0 0) 0x7f4f5813dc40 con 0x7f4f580b1f60 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 >> v1:172.21.15.203:6789/0 conn(0x7f4f58143d10 legacy=0x7f4f58138d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 >> v1:172.21.15.145:6789/0 conn(0x7f4f58139480 legacy=0x7f4f581e73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 monclient: found mon.c 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.276+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f58145ab0 con 0x7f4f580b1f60 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f4e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f4e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:42:58.286 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f5ffd6640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f5ffd6640 10 monclient: _renew_subs 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f5ffd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f58145d00 con 0x7f4f580b1f60 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f5ffd6640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f5ffd6640 10 monclient: _renew_subs 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f5ffd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f5ffd6640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f581df290 con 0x7f4f580b1f60 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.277+0000 7f4f5ffd6640 1 librados: init done 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.278+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f54001b70 con 0x7f4f580b1f60 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.278+0000 7f4f4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.278+0000 7f4f4ffff640 4 set_mon_vals no callback set 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.278+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4f54004c30 con 0x7f4f580b1f60 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.278+0000 7f4f4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.278+0000 7f4f4e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:42:58.287 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.279+0000 7f4f4e7fc640 10 monclient: dump: 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:42:58.288 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:42:58.289 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.280+0000 7f4f3bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:58.289 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.280+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f4f5401ac80 con 0x7f4f580b1f60 2024-03-29T11:42:58.289 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.280+0000 7f4f3bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:58.289 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.280+0000 7f4f3bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:58.289 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.280+0000 7f4f3bfff640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f58075f10 con 0x7f4f580b1f60 2024-03-29T11:42:58.289 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.281+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f4f54084c10 con 0x7f4f580b1f60 2024-03-29T11:42:58.290 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.281+0000 7f4f4e7fc640 10 client.4544.objecter ms_dispatch 0x7f4f58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:42:58.290 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.281+0000 7f4f4e7fc640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:42:58.290 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.281+0000 7f4f4e7fc640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:42:58.290 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.281+0000 7f4f4e7fc640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-29T11:42:58.290 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.284+0000 7f4f5d571640 10 monclient: get_auth_request con 0x7f4f3c06dd40 auth_method 0 2024-03-29T11:42:58.290 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.284+0000 7f4f4e7fc640 10 client.4544.objecter ms_handle_connect 0x7f4f3c06dd40 2024-03-29T11:42:58.293 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.290+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f54050320 con 0x7f4f580b1f60 2024-03-29T11:42:58.293 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.290+0000 7f4f4e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:42:58.293 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.290+0000 7f4f4e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:42:58.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.608+0000 7f4f3bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:58.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.608+0000 7f4f3bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:58.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.608+0000 7f4f3bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:42:58.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.608+0000 7f4f3bfff640 1 -- 172.21.15.145:0/2975830038 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f580b6bd0 con 0x7f4f580b1f60 2024-03-29T11:42:58.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.613+0000 7f4f4e7fc640 1 -- 172.21.15.145:0/2975830038 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f4f54052420 con 0x7f4f580b1f60 2024-03-29T11:42:58.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.613+0000 7f4f4e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:42:58.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.613+0000 7f4f4e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:42:58.621 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:42:58.622 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:42:58.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.618+0000 7f4f3bfff640 10 librados: watch_flush enter 2024-03-29T11:42:58.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 10 librados: watch_flush exit 2024-03-29T11:42:58.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-29T11:42:58.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 10 client.4544.objecter successfully canceled tick 2024-03-29T11:42:58.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 1 -- 172.21.15.145:0/2975830038 >> v1:172.21.15.145:6816/227372239 conn(0x7f4f3c06dd40 legacy=0x7f4f3c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:58.623 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 10 monclient: shutdown 2024-03-29T11:42:58.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:42:58.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 1 -- 172.21.15.145:0/2975830038 >> v1:172.21.15.145:6790/0 conn(0x7f4f580b1f60 legacy=0x7f4f5813c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:42:58.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 1 -- 172.21.15.145:0/2975830038 shutdown_connections 2024-03-29T11:42:58.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 1 -- 172.21.15.145:0/2975830038 >> 172.21.15.145:0/2975830038 conn(0x7f4f58066a00 msgr2=0x7f4f580b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:42:58.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 1 -- 172.21.15.145:0/2975830038 shutdown_connections 2024-03-29T11:42:58.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 1 -- 172.21.15.145:0/2975830038 wait complete. 2024-03-29T11:42:58.624 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:42:58.619+0000 7f4f3bfff640 1 librados: shutdown 2024-03-29T11:42:58.642 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:05.449 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:05.580 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.577+0000 7f9977b82640 10 monclient: get_monmap_and_config 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.577+0000 7f9977b82640 10 monclient: build_initial_monmap 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.577+0000 7f9977b82640 10 monclient: monmap: 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:43:05.578412+0000 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:43:05.578412+0000 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:05.581 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:43:05.582 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:43:05.582 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:05.582 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.577+0000 7f9977b82640 1 Processor -- start 2024-03-29T11:43:05.582 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.578+0000 7f9977b82640 1 -- start start 2024-03-29T11:43:05.582 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.578+0000 7f9977b82640 10 monclient: init 2024-03-29T11:43:05.586 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.578+0000 7f9977b82640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:05.586 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.578+0000 7f9977b82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f9977b82640 10 monclient(hunting): picked mon.noname-a con 0x7f99701422b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f9977b82640 10 monclient(hunting): picked mon.noname-b con 0x7f9970142690 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f9977b82640 10 monclient(hunting): picked mon.noname-c con 0x7f99700b17d0 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f9977b82640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9970142c80 con 0x7f99701422b0 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f9977b82640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99700aae40 con 0x7f99700b17d0 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f9977b82640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f997013fbd0 con 0x7f9970142690 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f9977b82640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f9977b82640 10 monclient(hunting): authenticate will time out at 1374.887138s 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f997611f640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f99700b17d0 0x7f99700b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:37356/0 (socket says 172.21.15.145:37356) 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f997611f640 1 -- 172.21.15.145:0/2440703343 learned_addr learned my addr 172.21.15.145:0/2440703343 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:05.587 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f997491c640 1 -- 172.21.15.145:0/2440703343 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154180715 0 0) 0x7f99700aae40 con 0x7f99700b17d0 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f997491c640 10 monclient(hunting): _negotiate 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f997491c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f997491c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f997491c640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.579+0000 7f997491c640 1 -- 172.21.15.145:0/2440703343 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f995c00bd30 con 0x7f99700b17d0 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.580+0000 7f997491c640 1 -- 172.21.15.145:0/2440703343 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3590072549 0 0) 0x7f995c00bd30 con 0x7f99700b17d0 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.580+0000 7f997491c640 1 -- 172.21.15.145:0/2440703343 >> v1:172.21.15.145:6790/0 conn(0x7f9970142690 legacy=0x7f99700b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.580+0000 7f997491c640 1 -- 172.21.15.145:0/2440703343 >> v1:172.21.15.145:6789/0 conn(0x7f99701422b0 legacy=0x7f997013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: _finish_hunting 0 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 1 monclient: found mon.noname-c 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:43:05.588 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 1 -- 172.21.15.145:0/2440703343 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9970151940 con 0x7f99700b17d0 2024-03-29T11:43:05.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: _finish_auth 0 2024-03-29T11:43:05.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: _check_auth_tickets 2024-03-29T11:43:05.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:05.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 1 -- 172.21.15.145:0/2440703343 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9968001a10 con 0x7f99700b17d0 2024-03-29T11:43:05.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:05.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:05.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: dump: 2024-03-29T11:43:05.589 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:05.589 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f9977b82640 5 monclient: authenticate success, global_id 4522 2024-03-29T11:43:05.590 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f9977b82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:05.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 1 -- 172.21.15.145:0/2440703343 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9968001c30 con 0x7f99700b17d0 2024-03-29T11:43:05.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:05.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 1 -- 172.21.15.145:0/2440703343 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9968001fc0 con 0x7f99700b17d0 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f997491c640 10 monclient: dump: 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f9977b82640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:05.599 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.582+0000 7f9977b82640 4 set_mon_vals no callback set 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.583+0000 7f9977b82640 10 monclient: shutdown 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.583+0000 7f9977b82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.583+0000 7f9977b82640 1 -- 172.21.15.145:0/2440703343 >> v1:172.21.15.203:6789/0 conn(0x7f99700b17d0 legacy=0x7f99700b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.584+0000 7f9977b82640 1 -- 172.21.15.145:0/2440703343 shutdown_connections 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.584+0000 7f9977b82640 1 -- 172.21.15.145:0/2440703343 >> 172.21.15.145:0/2440703343 conn(0x7f9970066a00 msgr2=0x7f9970066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.584+0000 7f9977b82640 1 -- 172.21.15.145:0/2440703343 shutdown_connections 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.584+0000 7f9977b82640 1 -- 172.21.15.145:0/2440703343 wait complete. 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.585+0000 7f9977b82640 10 monclient: build_initial_monmap 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.585+0000 7f9977b82640 10 monclient: monmap: 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:43:05.600 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.585+0000 7f9977b82640 1 librados: starting msgr at 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.585+0000 7f9977b82640 1 librados: starting objecter 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.586+0000 7f9977b82640 1 Processor -- start 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.586+0000 7f9977b82640 1 -- start start 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.586+0000 7f9977b82640 1 librados: setting wanted keys 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.586+0000 7f9977b82640 1 librados: calling monclient init 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.586+0000 7f9977b82640 10 monclient: init 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.586+0000 7f9977b82640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f9977b82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f9977b82640 10 monclient(hunting): picked mon.noname-a con 0x7f99700b17d0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:05.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f9977b82640 10 monclient(hunting): picked mon.noname-b con 0x7f99701422b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f9977b82640 10 monclient(hunting): picked mon.noname-c con 0x7f9970142690 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f9977b82640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9970070670 con 0x7f99700b17d0 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f9977b82640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99701d7600 con 0x7f99701422b0 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f9977b82640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99700b6990 con 0x7f9970142690 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f9977b82640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f9977b82640 10 monclient(hunting): authenticate will time out at 1374.895216s 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f997511d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f99701422b0 0x7f99701d6ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:37782/0 (socket says 172.21.15.145:37782) 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f997511d640 1 -- 172.21.15.145:0/2917998754 learned_addr learned my addr 172.21.15.145:0/2917998754 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f99627fc640 10 client.?.objecter ms_handle_connect 0x7f99701422b0 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f99627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f99627fc640 10 client.?.objecter ms_handle_connect 0x7f9970142690 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.587+0000 7f99627fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247117611 0 0) 0x7f99700b6990 con 0x7f9970142690 2024-03-29T11:43:05.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9954004eb0 con 0x7f9970142690 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2656769497 0 0) 0x7f9954004eb0 con 0x7f9970142690 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99700b6990 con 0x7f9970142690 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f99680018d0 con 0x7f9970142690 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 10 monclient(hunting): dump: 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:05.603 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.588+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2549128309 0 0) 0x7f99700b6990 con 0x7f9970142690 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.589+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 >> v1:172.21.15.203:6789/0 conn(0x7f99701422b0 legacy=0x7f99701d6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.589+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 >> v1:172.21.15.145:6789/0 conn(0x7f99700b17d0 legacy=0x7f99701d67e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 1 monclient: found mon.c 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9970070250 con 0x7f9970142690 2024-03-29T11:43:05.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f9977b82640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f9977b82640 10 monclient: _renew_subs 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f9977b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9970072010 con 0x7f9970142690 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f9977b82640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f9977b82640 10 monclient: _renew_subs 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f9977b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f997006f0b0 con 0x7f9970142690 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f9977b82640 1 librados: init done 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9968004e10 con 0x7f9970142690 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f99680015d0 con 0x7f9970142690 2024-03-29T11:43:05.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.590+0000 7f99627fc640 10 monclient: dump: 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.591+0000 7f9963fff640 4 set_mon_vals no callback set 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.592+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f996801af90 con 0x7f9970142690 2024-03-29T11:43:05.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.593+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f9968084e10 con 0x7f9970142690 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.593+0000 7f99627fc640 10 client.4550.objecter ms_dispatch 0x7f9970000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.593+0000 7f99627fc640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.593+0000 7f99627fc640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.593+0000 7f997591e640 10 monclient: get_auth_request con 0x7f995406de10 auth_method 0 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.593+0000 7f99627fc640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.593+0000 7f99627fc640 10 client.4550.objecter ms_handle_connect 0x7f995406de10 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.594+0000 7f9977b82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.594+0000 7f9977b82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.594+0000 7f9977b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:05.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.594+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9958005080 con 0x7f9970142690 2024-03-29T11:43:05.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.602+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f99680506f0 con 0x7f9970142690 2024-03-29T11:43:05.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.602+0000 7f99627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:05.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.602+0000 7f99627fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:05.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.899+0000 7f9977b82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:05.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.899+0000 7f9977b82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:05.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.899+0000 7f9977b82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:05.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.899+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9958005330 con 0x7f9970142690 2024-03-29T11:43:05.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.900+0000 7f99627fc640 1 -- 172.21.15.145:0/2917998754 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f9968052620 con 0x7f9970142690 2024-03-29T11:43:05.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.900+0000 7f99627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:05.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.900+0000 7f99627fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:05.904 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:43:05.905 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:05.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.904+0000 7f9977b82640 10 librados: watch_flush enter 2024-03-29T11:43:05.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.904+0000 7f9977b82640 10 librados: watch_flush exit 2024-03-29T11:43:05.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.904+0000 7f9977b82640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-29T11:43:05.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.904+0000 7f9977b82640 10 client.4550.objecter successfully canceled tick 2024-03-29T11:43:05.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.905+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 >> v1:172.21.15.145:6816/227372239 conn(0x7f995406de10 legacy=0x7f99540702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:05.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.905+0000 7f9977b82640 10 monclient: shutdown 2024-03-29T11:43:05.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.905+0000 7f9977b82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:05.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.905+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 >> v1:172.21.15.145:6790/0 conn(0x7f9970142690 legacy=0x7f99700b6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:05.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.905+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 shutdown_connections 2024-03-29T11:43:05.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.905+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 >> 172.21.15.145:0/2917998754 conn(0x7f9970066a00 msgr2=0x7f99700b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:05.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.905+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 shutdown_connections 2024-03-29T11:43:05.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.905+0000 7f9977b82640 1 -- 172.21.15.145:0/2917998754 wait complete. 2024-03-29T11:43:05.910 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:05.905+0000 7f9977b82640 1 librados: shutdown 2024-03-29T11:43:05.923 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:12.730 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:12.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.885+0000 7f5da938f640 10 monclient: get_monmap_and_config 2024-03-29T11:43:12.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.885+0000 7f5da938f640 10 monclient: build_initial_monmap 2024-03-29T11:43:12.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.885+0000 7f5da938f640 10 monclient: monmap: 2024-03-29T11:43:12.890 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:12.890 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:12.890 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:43:12.885847+0000 2024-03-29T11:43:12.890 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:43:12.885847+0000 2024-03-29T11:43:12.890 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:12.891 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:12.891 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:12.891 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:43:12.891 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:43:12.891 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:12.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.888+0000 7f5da938f640 1 Processor -- start 2024-03-29T11:43:12.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.888+0000 7f5da938f640 1 -- start start 2024-03-29T11:43:12.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.888+0000 7f5da938f640 10 monclient: init 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 10 monclient(hunting): picked mon.noname-a con 0x7f5da40af0f0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 10 monclient(hunting): picked mon.noname-b con 0x7f5da40af4d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 10 monclient(hunting): picked mon.noname-c con 0x7f5da40ab170 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da413f010 con 0x7f5da40af0f0 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40ab6c0 con 0x7f5da40ab170 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40ab9a0 con 0x7f5da40af4d0 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:12.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.889+0000 7f5da938f640 10 monclient(hunting): authenticate will time out at 1382.196181s 2024-03-29T11:43:12.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.894+0000 7f5da27fc640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f5da40af4d0 0x7f5da413f4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:57517/0 (socket says 172.21.15.145:57517) 2024-03-29T11:43:12.897 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.894+0000 7f5da27fc640 1 -- 172.21.15.145:0/3361262779 learned_addr learned my addr 172.21.15.145:0/3361262779 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:12.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.895+0000 7f5da1ffb640 1 -- 172.21.15.145:0/3361262779 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249101371 0 0) 0x7f5da40ab9a0 con 0x7f5da40af4d0 2024-03-29T11:43:12.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.895+0000 7f5da1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:43:12.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.895+0000 7f5da1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:12.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.895+0000 7f5da1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:12.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.895+0000 7f5da1ffb640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:43:12.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.895+0000 7f5da1ffb640 1 -- 172.21.15.145:0/3361262779 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d8c003520 con 0x7f5da40af4d0 2024-03-29T11:43:12.899 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.896+0000 7f5da1ffb640 1 -- 172.21.15.145:0/3361262779 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2863983295 0 0) 0x7f5d8c003520 con 0x7f5da40af4d0 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.896+0000 7f5da1ffb640 1 -- 172.21.15.145:0/3361262779 >> v1:172.21.15.203:6789/0 conn(0x7f5da40ab170 legacy=0x7f5da414a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.896+0000 7f5da1ffb640 1 -- 172.21.15.145:0/3361262779 >> v1:172.21.15.145:6789/0 conn(0x7f5da40af0f0 legacy=0x7f5da413c3d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 1 monclient: found mon.noname-b 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 1 -- 172.21.15.145:0/3361262779 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da40abc80 con 0x7f5da40af4d0 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 1 -- 172.21.15.145:0/3361262779 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5d9c001a10 con 0x7f5da40af4d0 2024-03-29T11:43:12.903 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.898+0000 7f5da1ffb640 10 monclient: dump: 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.899+0000 7f5da1ffb640 1 -- 172.21.15.145:0/3361262779 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d9c001c70 con 0x7f5da40af4d0 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.899+0000 7f5da1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.899+0000 7f5da1ffb640 1 -- 172.21.15.145:0/3361262779 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5d9c0020c0 con 0x7f5da40af4d0 2024-03-29T11:43:12.904 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.899+0000 7f5da1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.899+0000 7f5da1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.899+0000 7f5da1ffb640 10 monclient: dump: 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:12.905 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:12.906 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.900+0000 7f5da938f640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:43:12.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.900+0000 7f5da938f640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:12.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.901+0000 7f5da938f640 4 set_mon_vals no callback set 2024-03-29T11:43:12.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.901+0000 7f5da938f640 10 monclient: shutdown 2024-03-29T11:43:12.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.901+0000 7f5da938f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:12.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.901+0000 7f5da938f640 1 -- 172.21.15.145:0/3361262779 >> v1:172.21.15.145:6790/0 conn(0x7f5da40af4d0 legacy=0x7f5da413f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:12.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.903+0000 7f5da938f640 1 -- 172.21.15.145:0/3361262779 shutdown_connections 2024-03-29T11:43:12.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.903+0000 7f5da938f640 1 -- 172.21.15.145:0/3361262779 >> 172.21.15.145:0/3361262779 conn(0x7f5da40b35e0 msgr2=0x7f5da40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:12.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.904+0000 7f5da938f640 1 -- 172.21.15.145:0/3361262779 shutdown_connections 2024-03-29T11:43:12.907 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.904+0000 7f5da938f640 1 -- 172.21.15.145:0/3361262779 wait complete. 2024-03-29T11:43:12.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.905+0000 7f5da938f640 10 monclient: build_initial_monmap 2024-03-29T11:43:12.908 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.905+0000 7f5da938f640 10 monclient: monmap: 2024-03-29T11:43:12.908 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.905+0000 7f5da938f640 1 librados: starting msgr at 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.905+0000 7f5da938f640 1 librados: starting objecter 2024-03-29T11:43:12.909 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.905+0000 7f5da938f640 1 Processor -- start 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.906+0000 7f5da938f640 1 -- start start 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.906+0000 7f5da938f640 1 librados: setting wanted keys 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.906+0000 7f5da938f640 1 librados: calling monclient init 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.906+0000 7f5da938f640 10 monclient: init 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 10 monclient(hunting): picked mon.noname-a con 0x7f5da40ab170 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 10 monclient(hunting): picked mon.noname-b con 0x7f5da40af0f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 10 monclient(hunting): picked mon.noname-c con 0x7f5da41c7c10 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da41c5a20 con 0x7f5da40ab170 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da41c6fc0 con 0x7f5da40af0f0 2024-03-29T11:43:12.911 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da41df930 con 0x7f5da41c7c10 2024-03-29T11:43:12.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:12.912 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.907+0000 7f5da938f640 10 monclient(hunting): authenticate will time out at 1382.214296s 2024-03-29T11:43:12.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5da2ffd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5da40ab170 0x7f5da41c8360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:34015/0 (socket says 172.21.15.145:34015) 2024-03-29T11:43:12.915 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5da2ffd640 1 -- 172.21.15.145:0/1711134774 learned_addr learned my addr 172.21.15.145:0/1711134774 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:12.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 10 client.?.objecter ms_handle_connect 0x7f5da40ab170 2024-03-29T11:43:12.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:12.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 10 client.?.objecter ms_handle_connect 0x7f5da40af0f0 2024-03-29T11:43:12.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329369236 0 0) 0x7f5da41c5a20 con 0x7f5da40ab170 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.913+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d90004eb0 con 0x7f5da40ab170 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515565663 0 0) 0x7f5da41c6fc0 con 0x7f5da40af0f0 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d900051d0 con 0x7f5da40af0f0 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3526310634 0 0) 0x7f5d90004eb0 con 0x7f5da40ab170 2024-03-29T11:43:12.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5da41c6fc0 con 0x7f5da40ab170 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5d94001920 con 0x7f5da40ab170 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient(hunting): dump: 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:12.922 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 887542162 0 0) 0x7f5da41c6fc0 con 0x7f5da40ab170 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 >> v1:172.21.15.145:6790/0 conn(0x7f5da41c7c10 legacy=0x7f5da41df220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 >> v1:172.21.15.203:6789/0 conn(0x7f5da40af0f0 legacy=0x7f5da41deb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 monclient: found mon.a 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da41c72a0 con 0x7f5da40ab170 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5d8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5da938f640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5da938f640 10 monclient: _renew_subs 2024-03-29T11:43:12.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5da938f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.914+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da41c7580 con 0x7f5da40ab170 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.915+0000 7f5da938f640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.915+0000 7f5da938f640 10 monclient: _renew_subs 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.915+0000 7f5da938f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.915+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da41cb0d0 con 0x7f5da40ab170 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.915+0000 7f5da938f640 1 librados: init done 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.915+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d940035f0 con 0x7f5da40ab170 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.915+0000 7f5d8b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.915+0000 7f5da17fa640 4 set_mon_vals no callback set 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.917+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5d94001920 con 0x7f5da40ab170 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.917+0000 7f5d8b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.917+0000 7f5d8b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:12.924 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.917+0000 7f5d8b7fe640 10 monclient: dump: 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:12.925 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.917+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f5d940038e0 con 0x7f5da40ab170 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.918+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f5d940545c0 con 0x7f5da40ab170 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.918+0000 7f5d8b7fe640 10 client.4515.objecter ms_dispatch 0x7f5da4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.918+0000 7f5d8b7fe640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.918+0000 7f5d8b7fe640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.918+0000 7f5d8b7fe640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.919+0000 7f5da27fc640 10 monclient: get_auth_request con 0x7f5d9006e0d0 auth_method 0 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.919+0000 7f5d8b7fe640 10 client.4515.objecter ms_handle_connect 0x7f5d9006e0d0 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.920+0000 7f5da938f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.920+0000 7f5da938f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.920+0000 7f5da938f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:12.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.920+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d84005080 con 0x7f5da40ab170 2024-03-29T11:43:12.933 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.931+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d94050920 con 0x7f5da40ab170 2024-03-29T11:43:12.934 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.931+0000 7f5d8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:12.934 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:12.931+0000 7f5d8b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:13.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.258+0000 7f5da938f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:13.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.258+0000 7f5da938f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:13.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.258+0000 7f5da938f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:13.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.258+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d84005330 con 0x7f5da40ab170 2024-03-29T11:43:13.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.262+0000 7f5d8b7fe640 1 -- 172.21.15.145:0/1711134774 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f5d94050920 con 0x7f5da40ab170 2024-03-29T11:43:13.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.263+0000 7f5d8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:13.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.263+0000 7f5d8b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:13.266 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:43:13.267 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:13.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.267+0000 7f5da938f640 10 librados: watch_flush enter 2024-03-29T11:43:13.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.268+0000 7f5da938f640 10 librados: watch_flush exit 2024-03-29T11:43:13.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.268+0000 7f5da938f640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T11:43:13.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.268+0000 7f5da938f640 10 client.4515.objecter successfully canceled tick 2024-03-29T11:43:13.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.268+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 >> v1:172.21.15.145:6816/227372239 conn(0x7f5d9006e0d0 legacy=0x7f5d90070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:13.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.268+0000 7f5da938f640 10 monclient: shutdown 2024-03-29T11:43:13.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.268+0000 7f5da938f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:13.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.268+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 >> v1:172.21.15.145:6789/0 conn(0x7f5da40ab170 legacy=0x7f5da41c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:13.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.268+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 shutdown_connections 2024-03-29T11:43:13.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.268+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 >> 172.21.15.145:0/1711134774 conn(0x7f5da40b35e0 msgr2=0x7f5da413f4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:13.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.269+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 shutdown_connections 2024-03-29T11:43:13.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.269+0000 7f5da938f640 1 -- 172.21.15.145:0/1711134774 wait complete. 2024-03-29T11:43:13.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:13.269+0000 7f5da938f640 1 librados: shutdown 2024-03-29T11:43:13.289 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:20.096 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:20.244 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.240+0000 7fd98cafb640 10 monclient: get_monmap_and_config 2024-03-29T11:43:20.244 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.240+0000 7fd98cafb640 10 monclient: build_initial_monmap 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.240+0000 7fd98cafb640 10 monclient: monmap: 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:43:20.241916+0000 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:43:20.241916+0000 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:43:20.245 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:20.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.241+0000 7fd98cafb640 1 Processor -- start 2024-03-29T11:43:20.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.241+0000 7fd98cafb640 1 -- start start 2024-03-29T11:43:20.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.241+0000 7fd98cafb640 10 monclient: init 2024-03-29T11:43:20.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.242+0000 7fd98cafb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:20.246 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.242+0000 7fd98cafb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:43:20.247 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.243+0000 7fd98cafb640 10 monclient(hunting): picked mon.noname-b con 0x7fd9881457f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:20.247 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.243+0000 7fd98cafb640 10 monclient(hunting): picked mon.noname-a con 0x7fd98813f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:20.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.243+0000 7fd98cafb640 10 monclient(hunting): picked mon.noname-c con 0x7fd9880a9d70 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:20.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.243+0000 7fd98cafb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9880aaef0 con 0x7fd98813f650 2024-03-29T11:43:20.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.243+0000 7fd98cafb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd988145bd0 con 0x7fd9880a9d70 2024-03-29T11:43:20.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.243+0000 7fd98cafb640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd988145eb0 con 0x7fd9881457f0 2024-03-29T11:43:20.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.243+0000 7fd98cafb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.243+0000 7fd98cafb640 10 monclient(hunting): authenticate will time out at 1389.551556s 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.246+0000 7fd98659c640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fd9881457f0 0x7fd98813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:45071/0 (socket says 172.21.15.145:45071) 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.246+0000 7fd98659c640 1 -- 172.21.15.145:0/841484610 learned_addr learned my addr 172.21.15.145:0/841484610 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.246+0000 7fd98559a640 1 -- 172.21.15.145:0/841484610 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202186353 0 0) 0x7fd988145eb0 con 0x7fd9881457f0 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.246+0000 7fd98559a640 10 monclient(hunting): _negotiate 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.246+0000 7fd98559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.246+0000 7fd98559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.246+0000 7fd98559a640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.246+0000 7fd98559a640 1 -- 172.21.15.145:0/841484610 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd974003520 con 0x7fd9881457f0 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.247+0000 7fd98559a640 1 -- 172.21.15.145:0/841484610 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1223203769 0 0) 0x7fd974003520 con 0x7fd9881457f0 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.247+0000 7fd98559a640 1 -- 172.21.15.145:0/841484610 >> v1:172.21.15.203:6789/0 conn(0x7fd9880a9d70 legacy=0x7fd98814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:43:20.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.247+0000 7fd98559a640 1 -- 172.21.15.145:0/841484610 >> v1:172.21.15.145:6789/0 conn(0x7fd98813f650 legacy=0x7fd9880b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:20.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 10 monclient: _finish_hunting 0 2024-03-29T11:43:20.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 1 monclient: found mon.noname-b 2024-03-29T11:43:20.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:20.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:43:20.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 1 -- 172.21.15.145:0/841484610 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd988146340 con 0x7fd9881457f0 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 10 monclient: _finish_auth 0 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 10 monclient: _check_auth_tickets 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 1 -- 172.21.15.145:0/841484610 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd978001690 con 0x7fd9881457f0 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 10 monclient: dump: 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:20.256 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98cafb640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98cafb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.249+0000 7fd98559a640 1 -- 172.21.15.145:0/841484610 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9780019a0 con 0x7fd9881457f0 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.250+0000 7fd98559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.250+0000 7fd98559a640 1 -- 172.21.15.145:0/841484610 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd978001d30 con 0x7fd9881457f0 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.250+0000 7fd98559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.250+0000 7fd98559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.250+0000 7fd98559a640 10 monclient: dump: 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:20.257 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.250+0000 7fd98cafb640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.250+0000 7fd98cafb640 4 set_mon_vals no callback set 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.251+0000 7fd98cafb640 10 monclient: shutdown 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.251+0000 7fd98cafb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.251+0000 7fd98cafb640 1 -- 172.21.15.145:0/841484610 >> v1:172.21.15.145:6790/0 conn(0x7fd9881457f0 legacy=0x7fd98813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.251+0000 7fd98cafb640 1 -- 172.21.15.145:0/841484610 shutdown_connections 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.251+0000 7fd98cafb640 1 -- 172.21.15.145:0/841484610 >> 172.21.15.145:0/841484610 conn(0x7fd988066a00 msgr2=0x7fd988066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.251+0000 7fd98cafb640 1 -- 172.21.15.145:0/841484610 shutdown_connections 2024-03-29T11:43:20.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.251+0000 7fd98cafb640 1 -- 172.21.15.145:0/841484610 wait complete. 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.252+0000 7fd98cafb640 10 monclient: build_initial_monmap 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.252+0000 7fd98cafb640 10 monclient: monmap: 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:43:20.260 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.253+0000 7fd98cafb640 1 librados: starting msgr at 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.253+0000 7fd98cafb640 1 librados: starting objecter 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.253+0000 7fd98cafb640 1 Processor -- start 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.253+0000 7fd98cafb640 1 -- start start 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.253+0000 7fd98cafb640 1 librados: setting wanted keys 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.253+0000 7fd98cafb640 1 librados: calling monclient init 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.253+0000 7fd98cafb640 10 monclient: init 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.254+0000 7fd98cafb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.254+0000 7fd98cafb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.255+0000 7fd98cafb640 10 monclient(hunting): picked mon.noname-a con 0x7fd9880a9d70 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.255+0000 7fd98cafb640 10 monclient(hunting): picked mon.noname-b con 0x7fd98813f650 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.255+0000 7fd98cafb640 10 monclient(hunting): picked mon.noname-c con 0x7fd9881457f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.255+0000 7fd98cafb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98813c850 con 0x7fd9880a9d70 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.255+0000 7fd98cafb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9880b0db0 con 0x7fd98813f650 2024-03-29T11:43:20.261 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.255+0000 7fd98cafb640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9880b0f40 con 0x7fd9881457f0 2024-03-29T11:43:20.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.255+0000 7fd98cafb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:20.262 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.255+0000 7fd98cafb640 10 monclient(hunting): authenticate will time out at 1389.563929s 2024-03-29T11:43:20.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.259+0000 7fd986d9d640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fd9881457f0 0x7fd9881d6fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:45221/0 (socket says 172.21.15.145:45221) 2024-03-29T11:43:20.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.259+0000 7fd986d9d640 1 -- 172.21.15.145:0/703472580 learned_addr learned my addr 172.21.15.145:0/703472580 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:20.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 client.?.objecter ms_handle_connect 0x7fd9880a9d70 2024-03-29T11:43:20.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:20.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 client.?.objecter ms_handle_connect 0x7fd9881457f0 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017336667 0 0) 0x7fd98813c850 con 0x7fd9880a9d70 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd968004eb0 con 0x7fd9880a9d70 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 client.?.objecter ms_handle_connect 0x7fd98813f650 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 422792514 0 0) 0x7fd9880b0f40 con 0x7fd9881457f0 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:20.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:43:20.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd98813c850 con 0x7fd9881457f0 2024-03-29T11:43:20.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2108330492 0 0) 0x7fd9880b0db0 con 0x7fd98813f650 2024-03-29T11:43:20.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:20.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:20.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:20.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:43:20.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.260+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9880b0f40 con 0x7fd98813f650 2024-03-29T11:43:20.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.261+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2519642795 0 0) 0x7fd968004eb0 con 0x7fd9880a9d70 2024-03-29T11:43:20.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.261+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9880b0db0 con 0x7fd9880a9d70 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.261+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1525440895 0 0) 0x7fd98813c850 con 0x7fd9881457f0 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.261+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd968004eb0 con 0x7fd9881457f0 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.261+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd978001710 con 0x7fd9880a9d70 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.261+0000 7fd97effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 10 monclient(hunting): dump: 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:20.271 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3415757818 0 0) 0x7fd9880b0f40 con 0x7fd98813f650 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd98813c850 con 0x7fd98813f650 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd980004230 con 0x7fd9881457f0 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 10 monclient(hunting): dump: 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:20.272 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3402650700 0 0) 0x7fd9880b0db0 con 0x7fd9880a9d70 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 >> v1:172.21.15.145:6790/0 conn(0x7fd9881457f0 legacy=0x7fd9881d6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 >> v1:172.21.15.203:6789/0 conn(0x7fd98813f650 legacy=0x7fd988136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 1 monclient: found mon.a 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.262+0000 7fd97effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd97effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9880b10d0 con 0x7fd9880a9d70 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd97effd640 10 monclient: _finish_auth 0 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd97effd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd97effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:20.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd978004a90 con 0x7fd9880a9d70 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd98cafb640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd98cafb640 10 monclient: _renew_subs 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd98cafb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9881373d0 con 0x7fd9880a9d70 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd98cafb640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd98cafb640 10 monclient: _renew_subs 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd98cafb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9881d76f0 con 0x7fd9880a9d70 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.263+0000 7fd98cafb640 1 librados: init done 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.264+0000 7fd97effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.264+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fd978004e20 con 0x7fd9880a9d70 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.264+0000 7fd97effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.264+0000 7fd97effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:20.274 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.264+0000 7fd97effd640 10 monclient: dump: 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:20.275 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.265+0000 7fd98cafb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.265+0000 7fd98cafb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.265+0000 7fd98cafb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.265+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9881d46c0 con 0x7fd9880a9d70 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.265+0000 7fd984d99640 4 set_mon_vals no callback set 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.266+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fd97801adf0 con 0x7fd9880a9d70 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.267+0000 7fd985d9b640 10 monclient: get_auth_request con 0x7fd96806e290 auth_method 0 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.267+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fd978084a90 con 0x7fd9880a9d70 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.267+0000 7fd97effd640 10 client.4518.objecter ms_dispatch 0x7fd988000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.267+0000 7fd97effd640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.267+0000 7fd97effd640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:20.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.267+0000 7fd97effd640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-29T11:43:20.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.267+0000 7fd97effd640 10 client.4518.objecter ms_handle_connect 0x7fd96806e290 2024-03-29T11:43:20.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.274+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd978050370 con 0x7fd9880a9d70 2024-03-29T11:43:20.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.274+0000 7fd97effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:20.278 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.274+0000 7fd97effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:20.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.596+0000 7fd98cafb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:20.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.596+0000 7fd98cafb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:20.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.596+0000 7fd98cafb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:20.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.596+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd988075f10 con 0x7fd9880a9d70 2024-03-29T11:43:20.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.597+0000 7fd97effd640 1 -- 172.21.15.145:0/703472580 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fd9780522a0 con 0x7fd9880a9d70 2024-03-29T11:43:20.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.597+0000 7fd97effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:20.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.597+0000 7fd97effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:20.602 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:43:20.603 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:20.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.602+0000 7fd98cafb640 10 librados: watch_flush enter 2024-03-29T11:43:20.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.602+0000 7fd98cafb640 10 librados: watch_flush exit 2024-03-29T11:43:20.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.602+0000 7fd98cafb640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-29T11:43:20.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.602+0000 7fd98cafb640 10 client.4518.objecter successfully canceled tick 2024-03-29T11:43:20.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.603+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 >> v1:172.21.15.145:6816/227372239 conn(0x7fd96806e290 legacy=0x7fd968070750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:20.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.603+0000 7fd98cafb640 10 monclient: shutdown 2024-03-29T11:43:20.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.603+0000 7fd98cafb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:20.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.603+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 >> v1:172.21.15.145:6789/0 conn(0x7fd9880a9d70 legacy=0x7fd9881d3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:20.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.603+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 shutdown_connections 2024-03-29T11:43:20.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.603+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 >> 172.21.15.145:0/703472580 conn(0x7fd988066a00 msgr2=0x7fd988066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:20.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.604+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 shutdown_connections 2024-03-29T11:43:20.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.604+0000 7fd98cafb640 1 -- 172.21.15.145:0/703472580 wait complete. 2024-03-29T11:43:20.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:20.604+0000 7fd98cafb640 1 librados: shutdown 2024-03-29T11:43:20.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:27.435 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:27.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.584+0000 7f0aa2d3d640 10 monclient: get_monmap_and_config 2024-03-29T11:43:27.589 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.584+0000 7f0aa2d3d640 10 monclient: build_initial_monmap 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.584+0000 7f0aa2d3d640 10 monclient: monmap: 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:43:27.585561+0000 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:43:27.585561+0000 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:43:27.590 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:27.591 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.585+0000 7f0aa2d3d640 1 Processor -- start 2024-03-29T11:43:27.591 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.585+0000 7f0aa2d3d640 1 -- start start 2024-03-29T11:43:27.591 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.585+0000 7f0aa2d3d640 10 monclient: init 2024-03-29T11:43:27.591 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:27.591 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:27.591 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a9c0b0860 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:27.591 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a9c13ef30 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:27.591 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a9c0af8e0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:27.591 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c0b0c40 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c0aed10 con 0x7f0a9c13ef30 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c0afd50 con 0x7f0a9c0b0860 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.586+0000 7f0aa2d3d640 10 monclient(hunting): authenticate will time out at 1396.894293s 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.587+0000 7f0aa12da640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f0a9c0af8e0 0x7f0a9c149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:43180/0 (socket says 172.21.15.145:43180) 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.587+0000 7f0aa12da640 1 -- 172.21.15.145:0/4067081856 learned_addr learned my addr 172.21.15.145:0/4067081856 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.587+0000 7f0a9b7fe640 1 -- 172.21.15.145:0/4067081856 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890033973 0 0) 0x7f0a9c0b0c40 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.587+0000 7f0a9b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.587+0000 7f0a9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.587+0000 7f0a9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.587+0000 7f0a9b7fe640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:43:27.592 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.587+0000 7f0a9b7fe640 1 -- 172.21.15.145:0/4067081856 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a84003520 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.593 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.588+0000 7f0a9b7fe640 1 -- 172.21.15.145:0/4067081856 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4215931747 0 0) 0x7f0a84003520 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.593 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.588+0000 7f0a9b7fe640 1 -- 172.21.15.145:0/4067081856 >> v1:172.21.15.145:6790/0 conn(0x7f0a9c0b0860 legacy=0x7f0a9c13e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:27.593 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 1 -- 172.21.15.145:0/4067081856 >> v1:172.21.15.203:6789/0 conn(0x7f0a9c13ef30 legacy=0x7f0a9c0af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:27.594 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:27.594 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:43:27.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:27.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:43:27.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 1 -- 172.21.15.145:0/4067081856 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c0ab1b0 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:27.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:27.595 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 1 -- 172.21.15.145:0/4067081856 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0a8c001690 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: dump: 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:27.596 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0aa2d3d640 5 monclient: authenticate success, global_id 4521 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0aa2d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 1 -- 172.21.15.145:0/4067081856 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a8c001860 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 1 -- 172.21.15.145:0/4067081856 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0a8c001cb0 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0a9b7fe640 10 monclient: dump: 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:27.597 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0aa2d3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.590+0000 7f0aa2d3d640 4 set_mon_vals no callback set 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.591+0000 7f0aa2d3d640 10 monclient: shutdown 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.591+0000 7f0aa2d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.591+0000 7f0aa2d3d640 1 -- 172.21.15.145:0/4067081856 >> v1:172.21.15.145:6789/0 conn(0x7f0a9c0af8e0 legacy=0x7f0a9c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.591+0000 7f0aa2d3d640 1 -- 172.21.15.145:0/4067081856 shutdown_connections 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.591+0000 7f0aa2d3d640 1 -- 172.21.15.145:0/4067081856 >> 172.21.15.145:0/4067081856 conn(0x7f0a9c06b660 msgr2=0x7f0a9c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.591+0000 7f0aa2d3d640 1 -- 172.21.15.145:0/4067081856 shutdown_connections 2024-03-29T11:43:27.598 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.592+0000 7f0aa2d3d640 1 -- 172.21.15.145:0/4067081856 wait complete. 2024-03-29T11:43:27.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.593+0000 7f0aa2d3d640 10 monclient: build_initial_monmap 2024-03-29T11:43:27.600 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.593+0000 7f0aa2d3d640 10 monclient: monmap: 2024-03-29T11:43:27.600 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.593+0000 7f0aa2d3d640 1 librados: starting msgr at 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.593+0000 7f0aa2d3d640 1 librados: starting objecter 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.593+0000 7f0aa2d3d640 1 Processor -- start 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.594+0000 7f0aa2d3d640 1 -- start start 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.594+0000 7f0aa2d3d640 1 librados: setting wanted keys 2024-03-29T11:43:27.601 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.594+0000 7f0aa2d3d640 1 librados: calling monclient init 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.594+0000 7f0aa2d3d640 10 monclient: init 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.594+0000 7f0aa2d3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.594+0000 7f0aa2d3d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.595+0000 7f0aa2d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f0a9c0af8e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.595+0000 7f0aa2d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f0a9c0b0860 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.595+0000 7f0aa2d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f0a9c13ef30 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.595+0000 7f0aa2d3d640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c14a000 con 0x7f0a9c13ef30 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.595+0000 7f0aa2d3d640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c06e060 con 0x7f0a9c0b0860 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.595+0000 7f0aa2d3d640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a9c14a900 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.595+0000 7f0aa2d3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:27.602 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.595+0000 7f0aa2d3d640 10 monclient(hunting): authenticate will time out at 1396.903084s 2024-03-29T11:43:27.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.598+0000 7f0aa0ad9640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f0a9c0af8e0 0x7f0a9c148f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:54555/0 (socket says 172.21.15.145:54555) 2024-03-29T11:43:27.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.598+0000 7f0aa0ad9640 1 -- 172.21.15.145:0/325582476 learned_addr learned my addr 172.21.15.145:0/325582476 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:27.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 client.?.objecter ms_handle_connect 0x7f0a9c0af8e0 2024-03-29T11:43:27.604 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 client.?.objecter ms_handle_connect 0x7f0a9c13ef30 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786539049 0 0) 0x7f0a9c14a900 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a88004eb0 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4096024063 0 0) 0x7f0a9c14a000 con 0x7f0a9c13ef30 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a9c14a900 con 0x7f0a9c13ef30 2024-03-29T11:43:27.605 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.599+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2168652375 0 0) 0x7f0a88004eb0 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a9c14a000 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0a94004400 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 10 monclient(hunting): dump: 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:27.606 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4203235688 0 0) 0x7f0a9c14a900 con 0x7f0a9c13ef30 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a88004eb0 con 0x7f0a9c13ef30 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3794347541 0 0) 0x7f0a9c14a000 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 >> v1:172.21.15.203:6789/0 conn(0x7f0a9c0b0860 legacy=0x7f0a9c06e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 >> v1:172.21.15.145:6789/0 conn(0x7f0a9c13ef30 legacy=0x7f0a9c1498f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 1 monclient: found mon.c 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:27.607 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9c0b3d40 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:27.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:27.608 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0a997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.600+0000 7f0aa2d3d640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.601+0000 7f0aa2d3d640 10 monclient: _renew_subs 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.601+0000 7f0aa2d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.601+0000 7f0aa2d3d640 1 -- 172.21.15.145:0/325582476 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a9c135280 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.601+0000 7f0aa2d3d640 10 client.4562.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.601+0000 7f0aa2d3d640 10 monclient: _renew_subs 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.601+0000 7f0aa2d3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.601+0000 7f0aa2d3d640 1 -- 172.21.15.145:0/325582476 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a9c1e7590 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.601+0000 7f0aa2d3d640 1 librados: init done 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.602+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a94004560 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.602+0000 7f0a997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.602+0000 7f0a9a7fc640 4 set_mon_vals no callback set 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.603+0000 7f0a82ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.603+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0a94001500 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.609 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.603+0000 7f0a997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.603+0000 7f0a997fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.603+0000 7f0a997fa640 10 monclient: dump: 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.604+0000 7f0a82ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:27.610 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.604+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f0a94001660 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.611 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.604+0000 7f0a82ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:27.611 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.604+0000 7f0a82ffd640 1 -- 172.21.15.145:0/325582476 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c005080 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.611 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.606+0000 7f0a9bfff640 10 monclient: get_auth_request con 0x7f0a8806dfa0 auth_method 0 2024-03-29T11:43:27.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.607+0000 7f0a997fa640 10 client.4562.objecter ms_handle_connect 0x7f0a8806dfa0 2024-03-29T11:43:27.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.607+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f0a94084ce0 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.607+0000 7f0a997fa640 10 client.4562.objecter ms_dispatch 0x7f0a9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:27.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.607+0000 7f0a997fa640 3 client.4562.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:27.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.607+0000 7f0a997fa640 3 client.4562.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:27.612 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.607+0000 7f0a997fa640 20 client.4562.objecter dump_active .. 0 homeless 2024-03-29T11:43:27.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.613+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a940505c0 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.613+0000 7f0a997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:27.617 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.613+0000 7f0a997fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:27.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.932+0000 7f0a82ffd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:27.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.932+0000 7f0a82ffd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:27.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.932+0000 7f0a82ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:27.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.932+0000 7f0a82ffd640 1 -- 172.21.15.145:0/325582476 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a7c005330 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.935+0000 7f0a997fa640 1 -- 172.21.15.145:0/325582476 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f0a940524f0 con 0x7f0a9c0af8e0 2024-03-29T11:43:27.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.935+0000 7f0a997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:27.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.935+0000 7f0a997fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:27.939 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:43:27.939 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:27.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.938+0000 7f0a82ffd640 10 librados: watch_flush enter 2024-03-29T11:43:27.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.938+0000 7f0a82ffd640 10 librados: watch_flush exit 2024-03-29T11:43:27.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.938+0000 7f0a82ffd640 20 client.4562.objecter shutdown clearing up homeless session... 2024-03-29T11:43:27.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.938+0000 7f0a82ffd640 10 client.4562.objecter successfully canceled tick 2024-03-29T11:43:27.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.938+0000 7f0a82ffd640 1 -- 172.21.15.145:0/325582476 >> v1:172.21.15.145:6816/227372239 conn(0x7f0a8806dfa0 legacy=0x7f0a88070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:27.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.938+0000 7f0a82ffd640 10 monclient: shutdown 2024-03-29T11:43:27.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.938+0000 7f0a82ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:27.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.938+0000 7f0a82ffd640 1 -- 172.21.15.145:0/325582476 >> v1:172.21.15.145:6790/0 conn(0x7f0a9c0af8e0 legacy=0x7f0a9c148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:27.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.939+0000 7f0a82ffd640 1 -- 172.21.15.145:0/325582476 shutdown_connections 2024-03-29T11:43:27.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.939+0000 7f0a82ffd640 1 -- 172.21.15.145:0/325582476 >> 172.21.15.145:0/325582476 conn(0x7f0a9c06b660 msgr2=0x7f0a9c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:27.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.939+0000 7f0a82ffd640 1 -- 172.21.15.145:0/325582476 shutdown_connections 2024-03-29T11:43:27.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.939+0000 7f0a82ffd640 1 -- 172.21.15.145:0/325582476 wait complete. 2024-03-29T11:43:27.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:27.939+0000 7f0a82ffd640 1 librados: shutdown 2024-03-29T11:43:27.961 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:34.768 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:34.917 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.913+0000 7fe8c6bf6640 10 monclient: get_monmap_and_config 2024-03-29T11:43:34.917 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.913+0000 7fe8c6bf6640 10 monclient: build_initial_monmap 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.913+0000 7fe8c6bf6640 10 monclient: monmap: 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:43:34.915066+0000 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:43:34.915066+0000 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:43:34.918 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:34.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.914+0000 7fe8c6bf6640 1 Processor -- start 2024-03-29T11:43:34.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.915+0000 7fe8c6bf6640 1 -- start start 2024-03-29T11:43:34.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.915+0000 7fe8c6bf6640 10 monclient: init 2024-03-29T11:43:34.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:34.919 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:34.920 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 10 monclient(hunting): picked mon.noname-c con 0x7fe8c01422b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:34.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 10 monclient(hunting): picked mon.noname-b con 0x7fe8c0142690 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:34.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 10 monclient(hunting): picked mon.noname-a con 0x7fe8c00b17d0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:34.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c0142c80 con 0x7fe8c00b17d0 2024-03-29T11:43:34.921 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c00aae40 con 0x7fe8c01422b0 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c013fbd0 con 0x7fe8c0142690 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c6bf6640 10 monclient(hunting): authenticate will time out at 1404.224865s 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c4992640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe8c01422b0 0x7fe8c013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:52442/0 (socket says 172.21.15.145:52442) 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.916+0000 7fe8c4992640 1 -- 172.21.15.145:0/2220018579 learned_addr learned my addr 172.21.15.145:0/2220018579 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755904661 0 0) 0x7fe8c00aae40 con 0x7fe8c01422b0 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b0003520 con 0x7fe8c01422b0 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713108995 0 0) 0x7fe8c0142c80 con 0x7fe8c00b17d0 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:43:34.922 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:34.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:34.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:43:34.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c00aae40 con 0x7fe8c00b17d0 2024-03-29T11:43:34.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1491585976 0 0) 0x7fe8b0003520 con 0x7fe8c01422b0 2024-03-29T11:43:34.923 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.917+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 >> v1:172.21.15.145:6790/0 conn(0x7fe8c0142690 legacy=0x7fe8c00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:34.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.918+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 >> v1:172.21.15.145:6789/0 conn(0x7fe8c00b17d0 legacy=0x7fe8c00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:34.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:43:34.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 1 monclient: found mon.noname-c 2024-03-29T11:43:34.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:34.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:43:34.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c0151940 con 0x7fe8c01422b0 2024-03-29T11:43:34.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:43:34.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:43:34.926 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe8b40015c0 con 0x7fe8c01422b0 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8c6bf6640 5 monclient: authenticate success, global_id 4534 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8c6bf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 10 monclient: dump: 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:34.927 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8b40017e0 con 0x7fe8c01422b0 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.919+0000 7fe8bf7fe640 1 -- 172.21.15.145:0/2220018579 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe8b4001af0 con 0x7fe8c01422b0 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.920+0000 7fe8bf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.920+0000 7fe8bf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.920+0000 7fe8bf7fe640 10 monclient: dump: 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:34.928 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.920+0000 7fe8c6bf6640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.920+0000 7fe8c6bf6640 4 set_mon_vals no callback set 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.921+0000 7fe8c6bf6640 10 monclient: shutdown 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.921+0000 7fe8c6bf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.921+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/2220018579 >> v1:172.21.15.203:6789/0 conn(0x7fe8c01422b0 legacy=0x7fe8c013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.921+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/2220018579 shutdown_connections 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.921+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/2220018579 >> 172.21.15.145:0/2220018579 conn(0x7fe8c0066a00 msgr2=0x7fe8c0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.921+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/2220018579 shutdown_connections 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.921+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/2220018579 wait complete. 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.922+0000 7fe8c6bf6640 10 monclient: build_initial_monmap 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.922+0000 7fe8c6bf6640 10 monclient: monmap: 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:34.929 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:34.930 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:43:34.930 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:43:34.930 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:34.930 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:34.930 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:34.930 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:43:34.930 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:43:34.930 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:34.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.923+0000 7fe8c6bf6640 1 librados: starting msgr at 2024-03-29T11:43:34.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.923+0000 7fe8c6bf6640 1 librados: starting objecter 2024-03-29T11:43:34.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.923+0000 7fe8c6bf6640 1 Processor -- start 2024-03-29T11:43:34.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.923+0000 7fe8c6bf6640 1 -- start start 2024-03-29T11:43:34.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.923+0000 7fe8c6bf6640 1 librados: setting wanted keys 2024-03-29T11:43:34.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.923+0000 7fe8c6bf6640 1 librados: calling monclient init 2024-03-29T11:43:34.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.923+0000 7fe8c6bf6640 10 monclient: init 2024-03-29T11:43:34.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:34.937 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 10 monclient(hunting): picked mon.noname-b con 0x7fe8c00b17d0 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 10 monclient(hunting): picked mon.noname-a con 0x7fe8c01422b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c4992640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe8c00b17d0 0x7fe8c0150aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:52992/0 (socket says 172.21.15.145:52992) 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c4992640 1 -- 172.21.15.145:0/328659883 learned_addr learned my addr 172.21.15.145:0/328659883 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 10 monclient(hunting): picked mon.noname-c con 0x7fe8c0142690 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c012e900 con 0x7fe8c01422b0 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c01511b0 con 0x7fe8c00b17d0 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c006e730 con 0x7fe8c0142690 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8c6bf6640 10 monclient(hunting): authenticate will time out at 1404.232788s 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8bd7fa640 10 client.?.objecter ms_handle_connect 0x7fe8c00b17d0 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.924+0000 7fe8bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 10 client.?.objecter ms_handle_connect 0x7fe8c01422b0 2024-03-29T11:43:34.938 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674583630 0 0) 0x7fe8c012e900 con 0x7fe8c01422b0 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8a4004eb0 con 0x7fe8c01422b0 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988968047 0 0) 0x7fe8c01511b0 con 0x7fe8c00b17d0 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.925+0000 7fe8bd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8c012e900 con 0x7fe8c00b17d0 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1883321845 0 0) 0x7fe8a4004eb0 con 0x7fe8c01422b0 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8c01511b0 con 0x7fe8c01422b0 2024-03-29T11:43:34.939 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe8ac0017d0 con 0x7fe8c01422b0 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 10 monclient(hunting): dump: 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:34.940 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3617656003 0 0) 0x7fe8c01511b0 con 0x7fe8c01422b0 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.926+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 >> v1:172.21.15.145:6790/0 conn(0x7fe8c0142690 legacy=0x7fe8c012e1f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 >> v1:172.21.15.203:6789/0 conn(0x7fe8c00b17d0 legacy=0x7fe8c0150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 1 monclient: found mon.a 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c006ea10 con 0x7fe8c01422b0 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8c6bf6640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8c6bf6640 10 monclient: _renew_subs 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8c6bf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c006ecf0 con 0x7fe8c01422b0 2024-03-29T11:43:34.941 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8c6bf6640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8c6bf6640 10 monclient: _renew_subs 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8c6bf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c01dea40 con 0x7fe8c01422b0 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8c6bf6640 1 librados: init done 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8ac001ce0 con 0x7fe8c01422b0 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe8ac003790 con 0x7fe8c01422b0 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8bd7fa640 10 monclient: dump: 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:34.942 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.927+0000 7fe8beffd640 4 set_mon_vals no callback set 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.929+0000 7fe8c6bf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.929+0000 7fe8c6bf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.929+0000 7fe8c6bf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.929+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c0075f10 con 0x7fe8c01422b0 2024-03-29T11:43:34.943 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.930+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fe8ac019760 con 0x7fe8c01422b0 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.931+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fe8ac0836c0 con 0x7fe8c01422b0 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.931+0000 7fe8bd7fa640 10 client.4530.objecter ms_dispatch 0x7fe8c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.931+0000 7fe8bd7fa640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.931+0000 7fe8c4992640 10 monclient: get_auth_request con 0x7fe8a406dc60 auth_method 0 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.931+0000 7fe8bd7fa640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.931+0000 7fe8bd7fa640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.931+0000 7fe8bd7fa640 10 client.4530.objecter ms_handle_connect 0x7fe8a406dc60 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.938+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8ac04f000 con 0x7fe8c01422b0 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.938+0000 7fe8bd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:34.944 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:34.938+0000 7fe8bd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:35.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.251+0000 7fe8c6bf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:35.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.251+0000 7fe8c6bf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:35.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.251+0000 7fe8c6bf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:43:35.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.251+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8c00b2a30 con 0x7fe8c01422b0 2024-03-29T11:43:35.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.253+0000 7fe8bd7fa640 1 -- 172.21.15.145:0/328659883 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fe8ac050f30 con 0x7fe8c01422b0 2024-03-29T11:43:35.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.253+0000 7fe8bd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:35.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.253+0000 7fe8bd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:35.257 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:43:35.257 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:35.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 10 librados: watch_flush enter 2024-03-29T11:43:35.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 10 librados: watch_flush exit 2024-03-29T11:43:35.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T11:43:35.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 10 client.4530.objecter successfully canceled tick 2024-03-29T11:43:35.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 >> v1:172.21.15.145:6816/227372239 conn(0x7fe8a406dc60 legacy=0x7fe8a4070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:35.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 10 monclient: shutdown 2024-03-29T11:43:35.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:35.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 >> v1:172.21.15.145:6789/0 conn(0x7fe8c01422b0 legacy=0x7fe8c012dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:35.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 shutdown_connections 2024-03-29T11:43:35.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.258+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 >> 172.21.15.145:0/328659883 conn(0x7fe8c0066a00 msgr2=0x7fe8c013c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:35.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.259+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 shutdown_connections 2024-03-29T11:43:35.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.259+0000 7fe8c6bf6640 1 -- 172.21.15.145:0/328659883 wait complete. 2024-03-29T11:43:35.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:35.259+0000 7fe8c6bf6640 1 librados: shutdown 2024-03-29T11:43:35.280 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:42.085 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:42.247 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.243+0000 7f1ada855640 10 monclient: get_monmap_and_config 2024-03-29T11:43:42.247 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.243+0000 7f1ada855640 10 monclient: build_initial_monmap 2024-03-29T11:43:42.247 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.243+0000 7f1ada855640 10 monclient: monmap: 2024-03-29T11:43:42.247 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:42.247 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:42.248 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:43:42.244831+0000 2024-03-29T11:43:42.248 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:43:42.244831+0000 2024-03-29T11:43:42.248 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:42.248 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:42.248 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:42.248 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:43:42.248 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:43:42.248 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:42.249 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.245+0000 7f1ada855640 1 Processor -- start 2024-03-29T11:43:42.250 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.246+0000 7f1ada855640 1 -- start start 2024-03-29T11:43:42.250 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.246+0000 7f1ada855640 10 monclient: init 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.246+0000 7f1ada855640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.246+0000 7f1ada855640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.246+0000 7f1ada855640 10 monclient(hunting): picked mon.noname-c con 0x7f1ad41422b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.247+0000 7f1ada855640 10 monclient(hunting): picked mon.noname-b con 0x7f1ad4142690 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.247+0000 7f1ada855640 10 monclient(hunting): picked mon.noname-a con 0x7f1ad40b17d0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.247+0000 7f1ada855640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad4142c80 con 0x7f1ad40b17d0 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.247+0000 7f1ada855640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad40aae40 con 0x7f1ad41422b0 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.247+0000 7f1ada855640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad413fbd0 con 0x7f1ad4142690 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.247+0000 7f1ada855640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:42.251 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.247+0000 7f1ada855640 10 monclient(hunting): authenticate will time out at 1411.555081s 2024-03-29T11:43:42.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.247+0000 7f1ad8df2640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f1ad40b17d0 0x7f1ad40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:59284/0 (socket says 172.21.15.145:59284) 2024-03-29T11:43:42.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.247+0000 7f1ad8df2640 1 -- 172.21.15.145:0/3841481686 learned_addr learned my addr 172.21.15.145:0/3841481686 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:42.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972125882 0 0) 0x7f1ad4142c80 con 0x7f1ad40b17d0 2024-03-29T11:43:42.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:42.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:42.252 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac0003520 con 0x7f1ad40b17d0 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712002515 0 0) 0x7f1ad40aae40 con 0x7f1ad41422b0 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad4142c80 con 0x7f1ad41422b0 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810498088 0 0) 0x7f1ad413fbd0 con 0x7f1ad4142690 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:43:42.253 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.248+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad40aae40 con 0x7f1ad4142690 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150716454 0 0) 0x7f1ac0003520 con 0x7f1ad40b17d0 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 >> v1:172.21.15.145:6790/0 conn(0x7f1ad4142690 legacy=0x7f1ad40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 >> v1:172.21.15.203:6789/0 conn(0x7f1ad41422b0 legacy=0x7f1ad413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 1 monclient: found mon.noname-a 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad4151940 con 0x7f1ad40b17d0 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:42.255 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1ac4001a10 con 0x7f1ad40b17d0 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ad2ffd640 10 monclient: dump: 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ada855640 5 monclient: authenticate success, global_id 4533 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.249+0000 7f1ada855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:42.256 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.250+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ac4001b70 con 0x7f1ad40b17d0 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.250+0000 7f1ad2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.250+0000 7f1ad2ffd640 1 -- 172.21.15.145:0/3841481686 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1ac4001f40 con 0x7f1ad40b17d0 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.250+0000 7f1ad2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.250+0000 7f1ad2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.250+0000 7f1ad2ffd640 10 monclient: dump: 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:42.257 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:42.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.250+0000 7f1ada855640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:42.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.250+0000 7f1ada855640 4 set_mon_vals no callback set 2024-03-29T11:43:42.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.251+0000 7f1ada855640 10 monclient: shutdown 2024-03-29T11:43:42.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.251+0000 7f1ada855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:42.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.251+0000 7f1ada855640 1 -- 172.21.15.145:0/3841481686 >> v1:172.21.15.145:6789/0 conn(0x7f1ad40b17d0 legacy=0x7f1ad40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:42.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.251+0000 7f1ada855640 1 -- 172.21.15.145:0/3841481686 shutdown_connections 2024-03-29T11:43:42.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.251+0000 7f1ada855640 1 -- 172.21.15.145:0/3841481686 >> 172.21.15.145:0/3841481686 conn(0x7f1ad4066a00 msgr2=0x7f1ad4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:42.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.251+0000 7f1ada855640 1 -- 172.21.15.145:0/3841481686 shutdown_connections 2024-03-29T11:43:42.258 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.251+0000 7f1ada855640 1 -- 172.21.15.145:0/3841481686 wait complete. 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.252+0000 7f1ada855640 10 monclient: build_initial_monmap 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.252+0000 7f1ada855640 10 monclient: monmap: 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:42.263 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.252+0000 7f1ada855640 1 librados: starting msgr at 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.252+0000 7f1ada855640 1 librados: starting objecter 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.253+0000 7f1ada855640 1 Processor -- start 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.253+0000 7f1ada855640 1 -- start start 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.253+0000 7f1ada855640 1 librados: setting wanted keys 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.253+0000 7f1ada855640 1 librados: calling monclient init 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.253+0000 7f1ada855640 10 monclient: init 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 10 monclient(hunting): picked mon.noname-c con 0x7f1ad40b17d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 10 monclient(hunting): picked mon.noname-b con 0x7f1ad41422b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 10 monclient(hunting): picked mon.noname-a con 0x7f1ad4142690 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:42.264 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad4131560 con 0x7f1ad4142690 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad4131010 con 0x7f1ad41422b0 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad413bc10 con 0x7f1ad40b17d0 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ada855640 10 monclient(hunting): authenticate will time out at 1411.562408s 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ad37fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f1ad41422b0 0x7f1ad41deb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:39428/0 (socket says 172.21.15.145:39428) 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.254+0000 7f1ad37fe640 1 -- 172.21.15.145:0/1026228659 learned_addr learned my addr 172.21.15.145:0/1026228659 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 client.?.objecter ms_handle_connect 0x7f1ad41422b0 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 client.?.objecter ms_handle_connect 0x7f1ad40b17d0 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 client.?.objecter ms_handle_connect 0x7f1ad4142690 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723747694 0 0) 0x7f1ad413bc10 con 0x7f1ad40b17d0 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:42.265 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab4004eb0 con 0x7f1ad40b17d0 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852836286 0 0) 0x7f1ad4131010 con 0x7f1ad41422b0 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.255+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad413bc10 con 0x7f1ad41422b0 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 127426511 0 0) 0x7f1ad413bc10 con 0x7f1ad41422b0 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ad4131010 con 0x7f1ad41422b0 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587574162 0 0) 0x7f1ad4131560 con 0x7f1ad4142690 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:43:42.266 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad413bc10 con 0x7f1ad4142690 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1abc001810 con 0x7f1ad41422b0 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:42.267 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 729034139 0 0) 0x7f1ab4004eb0 con 0x7f1ad40b17d0 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ad4131560 con 0x7f1ad40b17d0 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1acc0016b0 con 0x7f1ad40b17d0 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient(hunting): dump: 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:42.268 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2547236019 0 0) 0x7f1ad4131560 con 0x7f1ad40b17d0 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 >> v1:172.21.15.203:6789/0 conn(0x7f1ad41422b0 legacy=0x7f1ad41deb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 >> v1:172.21.15.145:6789/0 conn(0x7f1ad4142690 legacy=0x7f1ad4130900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 monclient: found mon.c 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad413bda0 con 0x7f1ad40b17d0 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:43:42.269 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.256+0000 7f1ad0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.257+0000 7f1ada855640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.257+0000 7f1ada855640 10 monclient: _renew_subs 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.257+0000 7f1ada855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.257+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad41e70a0 con 0x7f1ad40b17d0 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.257+0000 7f1ada855640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.257+0000 7f1ada855640 10 monclient: _renew_subs 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.257+0000 7f1ada855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.257+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad41e7410 con 0x7f1ad40b17d0 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.257+0000 7f1ada855640 1 librados: init done 2024-03-29T11:43:42.270 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.258+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1acc001a50 con 0x7f1ad40b17d0 2024-03-29T11:43:42.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.258+0000 7f1ad0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:42.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.258+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1acc0037d0 con 0x7f1ad40b17d0 2024-03-29T11:43:42.271 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.258+0000 7f1ad0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.258+0000 7f1ad0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.258+0000 7f1ad0ff9640 10 monclient: dump: 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.258+0000 7f1ad1ffb640 4 set_mon_vals no callback set 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.261+0000 7f1ada855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:42.272 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.261+0000 7f1ada855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.261+0000 7f1ada855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.261+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab8005080 con 0x7f1ad40b17d0 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.262+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f1acc0197a0 con 0x7f1ad40b17d0 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.263+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f1acc083860 con 0x7f1ad40b17d0 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.263+0000 7f1ad0ff9640 10 client.4568.objecter ms_dispatch 0x7f1ad4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.263+0000 7f1ad0ff9640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.263+0000 7f1ad0ff9640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.263+0000 7f1ad37fe640 10 monclient: get_auth_request con 0x7f1ab406df80 auth_method 0 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.264+0000 7f1ad0ff9640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-29T11:43:42.273 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.264+0000 7f1ad0ff9640 10 client.4568.objecter ms_handle_connect 0x7f1ab406df80 2024-03-29T11:43:42.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.273+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1acc04f1a0 con 0x7f1ad40b17d0 2024-03-29T11:43:42.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.273+0000 7f1ad0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:42.277 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.273+0000 7f1ad0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:42.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.671+0000 7f1ada855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:42.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.671+0000 7f1ada855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:42.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.671+0000 7f1ada855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:42.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.671+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ab8005330 con 0x7f1ad40b17d0 2024-03-29T11:43:42.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.673+0000 7f1ad0ff9640 1 -- 172.21.15.145:0/1026228659 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f1acc0510d0 con 0x7f1ad40b17d0 2024-03-29T11:43:42.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.673+0000 7f1ad0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:42.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.673+0000 7f1ad0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:42.678 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:43:42.678 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 10 librados: watch_flush enter 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 10 librados: watch_flush exit 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 10 client.4568.objecter successfully canceled tick 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 >> v1:172.21.15.145:6816/227372239 conn(0x7f1ab406df80 legacy=0x7f1ab4070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 10 monclient: shutdown 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 >> v1:172.21.15.145:6790/0 conn(0x7f1ad40b17d0 legacy=0x7f1ad41319f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 shutdown_connections 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 >> 172.21.15.145:0/1026228659 conn(0x7f1ad4066a00 msgr2=0x7f1ad413c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:42.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.677+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 shutdown_connections 2024-03-29T11:43:42.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.678+0000 7f1ada855640 1 -- 172.21.15.145:0/1026228659 wait complete. 2024-03-29T11:43:42.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:42.678+0000 7f1ada855640 1 librados: shutdown 2024-03-29T11:43:42.697 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:49.505 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:49.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.647+0000 7fa860cfb640 10 monclient: get_monmap_and_config 2024-03-29T11:43:49.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.647+0000 7fa860cfb640 10 monclient: build_initial_monmap 2024-03-29T11:43:49.651 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.647+0000 7fa860cfb640 10 monclient: monmap: 2024-03-29T11:43:49.651 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:49.651 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:49.651 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:43:49.648561+0000 2024-03-29T11:43:49.651 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:43:49.648561+0000 2024-03-29T11:43:49.652 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:49.652 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:49.652 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:49.652 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:43:49.652 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:43:49.652 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:49.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.648+0000 7fa860cfb640 1 Processor -- start 2024-03-29T11:43:49.653 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.648+0000 7fa860cfb640 1 -- start start 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.648+0000 7fa860cfb640 10 monclient: init 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 10 monclient(hunting): picked mon.noname-a con 0x7fa85c1422b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 10 monclient(hunting): picked mon.noname-c con 0x7fa85c142690 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 10 monclient(hunting): picked mon.noname-b con 0x7fa85c0b17d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c142c80 con 0x7fa85c1422b0 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c0aae40 con 0x7fa85c142690 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c13fbd0 con 0x7fa85c0b17d0 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:49.654 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.649+0000 7fa860cfb640 10 monclient(hunting): authenticate will time out at 1418.957916s 2024-03-29T11:43:49.655 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.650+0000 7fa85ad9d640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fa85c0b17d0 0x7fa85c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:44276/0 (socket says 172.21.15.145:44276) 2024-03-29T11:43:49.655 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.650+0000 7fa85ad9d640 1 -- 172.21.15.145:0/1734409528 learned_addr learned my addr 172.21.15.145:0/1734409528 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:49.655 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.651+0000 7fa859d9b640 1 -- 172.21.15.145:0/1734409528 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038619413 0 0) 0x7fa85c13fbd0 con 0x7fa85c0b17d0 2024-03-29T11:43:49.655 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.651+0000 7fa859d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:43:49.655 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.651+0000 7fa859d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:49.655 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.651+0000 7fa859d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:49.655 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.651+0000 7fa859d9b640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:43:49.655 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.651+0000 7fa859d9b640 1 -- 172.21.15.145:0/1734409528 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa848003520 con 0x7fa85c0b17d0 2024-03-29T11:43:49.655 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.651+0000 7fa859d9b640 1 -- 172.21.15.145:0/1734409528 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2677567534 0 0) 0x7fa848003520 con 0x7fa85c0b17d0 2024-03-29T11:43:49.656 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.651+0000 7fa859d9b640 1 -- 172.21.15.145:0/1734409528 >> v1:172.21.15.203:6789/0 conn(0x7fa85c142690 legacy=0x7fa85c0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:49.657 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 1 -- 172.21.15.145:0/1734409528 >> v1:172.21.15.145:6789/0 conn(0x7fa85c1422b0 legacy=0x7fa85c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:49.657 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 1 monclient: found mon.noname-b 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 1 -- 172.21.15.145:0/1734409528 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c151940 con 0x7fa85c0b17d0 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 10 monclient: _finish_auth 0 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa860cfb640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa860cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 1 -- 172.21.15.145:0/1734409528 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa854001a10 con 0x7fa85c0b17d0 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 10 monclient: dump: 2024-03-29T11:43:49.658 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa860cfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 1 -- 172.21.15.145:0/1734409528 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa854001cb0 con 0x7fa85c0b17d0 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.653+0000 7fa859d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.654+0000 7fa860cfb640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:49.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.654+0000 7fa860cfb640 4 set_mon_vals no callback set 2024-03-29T11:43:49.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.654+0000 7fa859d9b640 1 -- 172.21.15.145:0/1734409528 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa854002080 con 0x7fa85c0b17d0 2024-03-29T11:43:49.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.654+0000 7fa859d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:49.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.654+0000 7fa859d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:49.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.654+0000 7fa859d9b640 10 monclient: dump: 2024-03-29T11:43:49.660 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:49.660 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:49.660 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:49.660 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:49.660 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.655+0000 7fa860cfb640 10 monclient: shutdown 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.655+0000 7fa860cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.655+0000 7fa860cfb640 1 -- 172.21.15.145:0/1734409528 >> v1:172.21.15.145:6790/0 conn(0x7fa85c0b17d0 legacy=0x7fa85c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.655+0000 7fa860cfb640 1 -- 172.21.15.145:0/1734409528 shutdown_connections 2024-03-29T11:43:49.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.655+0000 7fa860cfb640 1 -- 172.21.15.145:0/1734409528 >> 172.21.15.145:0/1734409528 conn(0x7fa85c066a00 msgr2=0x7fa85c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:49.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.661+0000 7fa860cfb640 1 -- 172.21.15.145:0/1734409528 shutdown_connections 2024-03-29T11:43:49.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.661+0000 7fa860cfb640 1 -- 172.21.15.145:0/1734409528 wait complete. 2024-03-29T11:43:49.666 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.662+0000 7fa860cfb640 10 monclient: build_initial_monmap 2024-03-29T11:43:49.666 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.662+0000 7fa860cfb640 10 monclient: monmap: 2024-03-29T11:43:49.666 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:49.666 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:49.666 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:43:49.666 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:43:49.666 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:49.667 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:49.667 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:49.667 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:43:49.667 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:43:49.667 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.663+0000 7fa860cfb640 1 librados: starting msgr at 2024-03-29T11:43:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.663+0000 7fa860cfb640 1 librados: starting objecter 2024-03-29T11:43:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.663+0000 7fa860cfb640 1 Processor -- start 2024-03-29T11:43:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.663+0000 7fa860cfb640 1 -- start start 2024-03-29T11:43:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.663+0000 7fa860cfb640 1 librados: setting wanted keys 2024-03-29T11:43:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.663+0000 7fa860cfb640 1 librados: calling monclient init 2024-03-29T11:43:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.663+0000 7fa860cfb640 10 monclient: init 2024-03-29T11:43:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.664+0000 7fa860cfb640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:49.668 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.664+0000 7fa860cfb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:43:49.669 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.664+0000 7fa860cfb640 10 monclient(hunting): picked mon.noname-b con 0x7fa85c0b17d0 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.666+0000 7fa860cfb640 10 monclient(hunting): picked mon.noname-a con 0x7fa85c1422b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.666+0000 7fa860cfb640 10 monclient(hunting): picked mon.noname-c con 0x7fa85c142690 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:49.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.666+0000 7fa860cfb640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c071a20 con 0x7fa85c1422b0 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.666+0000 7fa860cfb640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c071c70 con 0x7fa85c0b17d0 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.666+0000 7fa860cfb640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa85c06ea90 con 0x7fa85c142690 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.666+0000 7fa860cfb640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.666+0000 7fa860cfb640 10 monclient(hunting): authenticate will time out at 1418.974700s 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa85a59c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fa85c0b17d0 0x7fa85c1deb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:34653/0 (socket says 172.21.15.145:34653) 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa85a59c640 1 -- 172.21.15.145:0/4055513585 learned_addr learned my addr 172.21.15.145:0/4055513585 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa852ffd640 10 client.?.objecter ms_handle_connect 0x7fa85c0b17d0 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa852ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143655965 0 0) 0x7fa85c071c70 con 0x7fa85c0b17d0 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa852ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa852ffd640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:43:49.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.667+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa83c004eb0 con 0x7fa85c0b17d0 2024-03-29T11:43:49.674 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 client.?.objecter ms_handle_connect 0x7fa85c142690 2024-03-29T11:43:49.674 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:49.674 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 215479474 0 0) 0x7fa85c06ea90 con 0x7fa85c142690 2024-03-29T11:43:49.674 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:43:49.674 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:49.674 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:49.674 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa85c071c70 con 0x7fa85c142690 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 32203633 0 0) 0x7fa83c004eb0 con 0x7fa85c0b17d0 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa85c06ea90 con 0x7fa85c0b17d0 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa84c001910 con 0x7fa85c0b17d0 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient(hunting): dump: 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:49.675 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1199552421 0 0) 0x7fa85c06ea90 con 0x7fa85c0b17d0 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 >> v1:172.21.15.145:6790/0 conn(0x7fa85c142690 legacy=0x7fa85c1e6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 >> v1:172.21.15.145:6789/0 conn(0x7fa85c1422b0 legacy=0x7fa85c12ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 monclient: found mon.b 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c06ed00 con 0x7fa85c0b17d0 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient: _finish_auth 0 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:43:49.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa852ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa860cfb640 5 monclient: authenticate success, global_id 4546 2024-03-29T11:43:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa860cfb640 10 monclient: _renew_subs 2024-03-29T11:43:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa860cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:43:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c06de90 con 0x7fa85c0b17d0 2024-03-29T11:43:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa860cfb640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa860cfb640 10 monclient: _renew_subs 2024-03-29T11:43:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa860cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:43:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c131a30 con 0x7fa85c0b17d0 2024-03-29T11:43:49.677 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.670+0000 7fa860cfb640 1 librados: init done 2024-03-29T11:43:49.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.672+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa84c003610 con 0x7fa85c0b17d0 2024-03-29T11:43:49.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.672+0000 7fa852ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:49.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.672+0000 7fa858d99640 4 set_mon_vals no callback set 2024-03-29T11:43:49.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.672+0000 7fa860cfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:49.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.673+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fa84c003a60 con 0x7fa85c0b17d0 2024-03-29T11:43:49.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.673+0000 7fa852ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:49.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.673+0000 7fa852ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:43:49.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.673+0000 7fa852ffd640 10 monclient: dump: 2024-03-29T11:43:49.678 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.673+0000 7fa860cfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.673+0000 7fa860cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.673+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa840005080 con 0x7fa85c0b17d0 2024-03-29T11:43:49.679 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.673+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fa84c003c80 con 0x7fa85c0b17d0 2024-03-29T11:43:49.680 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.674+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fa84c084940 con 0x7fa85c0b17d0 2024-03-29T11:43:49.680 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.675+0000 7fa853fff640 10 monclient: get_auth_request con 0x7fa83c06dcb0 auth_method 0 2024-03-29T11:43:49.680 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.676+0000 7fa852ffd640 10 client.4546.objecter ms_dispatch 0x7fa85c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:49.680 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.676+0000 7fa852ffd640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.676+0000 7fa852ffd640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.676+0000 7fa852ffd640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-29T11:43:49.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.676+0000 7fa852ffd640 10 client.4546.objecter ms_handle_connect 0x7fa83c06dcb0 2024-03-29T11:43:49.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.683+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa84c050220 con 0x7fa85c0b17d0 2024-03-29T11:43:49.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.683+0000 7fa852ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:49.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:49.683+0000 7fa852ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:50.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.014+0000 7fa860cfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:50.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.015+0000 7fa860cfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:50.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.015+0000 7fa860cfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:43:50.019 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.015+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa840005330 con 0x7fa85c0b17d0 2024-03-29T11:43:50.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.017+0000 7fa852ffd640 1 -- 172.21.15.145:0/4055513585 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fa84c052150 con 0x7fa85c0b17d0 2024-03-29T11:43:50.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.017+0000 7fa852ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:50.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.017+0000 7fa852ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:50.022 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:43:50.022 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:50.025 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.021+0000 7fa860cfb640 10 librados: watch_flush enter 2024-03-29T11:43:50.025 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.021+0000 7fa860cfb640 10 librados: watch_flush exit 2024-03-29T11:43:50.025 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.021+0000 7fa860cfb640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-29T11:43:50.025 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.021+0000 7fa860cfb640 10 client.4546.objecter successfully canceled tick 2024-03-29T11:43:50.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.022+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 >> v1:172.21.15.145:6816/227372239 conn(0x7fa83c06dcb0 legacy=0x7fa83c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:50.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.022+0000 7fa860cfb640 10 monclient: shutdown 2024-03-29T11:43:50.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.022+0000 7fa860cfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:50.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.022+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 >> v1:172.21.15.203:6789/0 conn(0x7fa85c0b17d0 legacy=0x7fa85c1deb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:50.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.022+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 shutdown_connections 2024-03-29T11:43:50.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.022+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 >> 172.21.15.145:0/4055513585 conn(0x7fa85c066a00 msgr2=0x7fa85c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:50.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.022+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 shutdown_connections 2024-03-29T11:43:50.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.022+0000 7fa860cfb640 1 -- 172.21.15.145:0/4055513585 wait complete. 2024-03-29T11:43:50.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:50.022+0000 7fa860cfb640 1 librados: shutdown 2024-03-29T11:43:50.040 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:43:56.848 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:43:57.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:56.998+0000 7f863c8c1640 10 monclient: get_monmap_and_config 2024-03-29T11:43:57.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:56.998+0000 7f863c8c1640 10 monclient: build_initial_monmap 2024-03-29T11:43:57.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:56.998+0000 7f863c8c1640 10 monclient: monmap: 2024-03-29T11:43:57.002 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:57.002 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:57.002 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:43:56.998655+0000 2024-03-29T11:43:57.003 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:43:56.998655+0000 2024-03-29T11:43:57.003 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:57.003 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:57.003 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:57.003 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:43:57.003 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:43:57.003 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:57.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:56.999+0000 7f863c8c1640 1 Processor -- start 2024-03-29T11:43:57.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:56.999+0000 7f863c8c1640 1 -- start start 2024-03-29T11:43:57.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:56.999+0000 7f863c8c1640 10 monclient: init 2024-03-29T11:43:57.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:57.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:43:57.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f86380b2800 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:57.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f86380b3220 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:57.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f86381439c0 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:57.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86380b3840 con 0x7f86381439c0 2024-03-29T11:43:57.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863806f230 con 0x7f86380b2800 2024-03-29T11:43:57.006 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863806ef50 con 0x7f86380b3220 2024-03-29T11:43:57.006 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:57.006 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.000+0000 7f863c8c1640 10 monclient(hunting): authenticate will time out at 1426.307865s 2024-03-29T11:43:57.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.004+0000 7f8637fff640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f86381439c0 0x7f8638151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:54855/0 (socket says 172.21.15.145:54855) 2024-03-29T11:43:57.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.004+0000 7f8637fff640 1 -- 172.21.15.145:0/2893540714 learned_addr learned my addr 172.21.15.145:0/2893540714 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.005+0000 7f86367fc640 1 -- 172.21.15.145:0/2893540714 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646735423 0 0) 0x7f86380b3840 con 0x7f86381439c0 2024-03-29T11:43:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.005+0000 7f86367fc640 10 monclient(hunting): _negotiate 2024-03-29T11:43:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.005+0000 7f86367fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.005+0000 7f86367fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.005+0000 7f86367fc640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:43:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.005+0000 7f86367fc640 1 -- 172.21.15.145:0/2893540714 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8620003520 con 0x7f86381439c0 2024-03-29T11:43:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.006+0000 7f86367fc640 1 -- 172.21.15.145:0/2893540714 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3309964223 0 0) 0x7f8620003520 con 0x7f86381439c0 2024-03-29T11:43:57.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.006+0000 7f86367fc640 1 -- 172.21.15.145:0/2893540714 >> v1:172.21.15.145:6790/0 conn(0x7f86380b3220 legacy=0x7f8638146710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:43:57.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.006+0000 7f86367fc640 1 -- 172.21.15.145:0/2893540714 >> v1:172.21.15.203:6789/0 conn(0x7f86380b2800 legacy=0x7f86381432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:57.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.006+0000 7f86367fc640 10 monclient: _finish_hunting 0 2024-03-29T11:43:57.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.006+0000 7f86367fc640 1 monclient: found mon.noname-a 2024-03-29T11:43:57.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.006+0000 7f86367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:57.015 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f86367fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:43:57.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f86367fc640 1 -- 172.21.15.145:0/2893540714 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8638071d30 con 0x7f86381439c0 2024-03-29T11:43:57.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f86367fc640 10 monclient: _finish_auth 0 2024-03-29T11:43:57.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f86367fc640 10 monclient: _check_auth_tickets 2024-03-29T11:43:57.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f86367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:57.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f86367fc640 1 -- 172.21.15.145:0/2893540714 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f862c001690 con 0x7f86381439c0 2024-03-29T11:43:57.016 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f86367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f86367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f86367fc640 10 monclient: dump: 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:57.017 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f863c8c1640 5 monclient: authenticate success, global_id 4542 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.007+0000 7f863c8c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.009+0000 7f86367fc640 1 -- 172.21.15.145:0/2893540714 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f862c001860 con 0x7f86381439c0 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.009+0000 7f86367fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.009+0000 7f863c8c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.009+0000 7f863c8c1640 4 set_mon_vals no callback set 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.010+0000 7f863c8c1640 10 monclient: shutdown 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.010+0000 7f863c8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.010+0000 7f863c8c1640 1 -- 172.21.15.145:0/2893540714 >> v1:172.21.15.145:6789/0 conn(0x7f86381439c0 legacy=0x7f8638151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.011+0000 7f863c8c1640 1 -- 172.21.15.145:0/2893540714 shutdown_connections 2024-03-29T11:43:57.018 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.011+0000 7f863c8c1640 1 -- 172.21.15.145:0/2893540714 >> 172.21.15.145:0/2893540714 conn(0x7f8638066a00 msgr2=0x7f8638066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:57.020 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.014+0000 7f863c8c1640 1 -- 172.21.15.145:0/2893540714 shutdown_connections 2024-03-29T11:43:57.020 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.014+0000 7f863c8c1640 1 -- 172.21.15.145:0/2893540714 wait complete. 2024-03-29T11:43:57.020 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.015+0000 7f863c8c1640 10 monclient: build_initial_monmap 2024-03-29T11:43:57.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.015+0000 7f863c8c1640 10 monclient: monmap: 2024-03-29T11:43:57.021 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:43:57.021 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:43:57.021 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:43:57.021 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:43:57.021 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:43:57.021 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:57.021 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:43:57.021 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:43:57.022 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:43:57.022 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.016+0000 7f863c8c1640 1 librados: starting msgr at 2024-03-29T11:43:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.016+0000 7f863c8c1640 1 librados: starting objecter 2024-03-29T11:43:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.016+0000 7f863c8c1640 1 Processor -- start 2024-03-29T11:43:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.016+0000 7f863c8c1640 1 -- start start 2024-03-29T11:43:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.016+0000 7f863c8c1640 1 librados: setting wanted keys 2024-03-29T11:43:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.016+0000 7f863c8c1640 1 librados: calling monclient init 2024-03-29T11:43:57.022 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.016+0000 7f863c8c1640 10 monclient: init 2024-03-29T11:43:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:43:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:43:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 10 monclient(hunting): picked mon.noname-a con 0x7f86380b2800 addr v1:172.21.15.145:6789/0 2024-03-29T11:43:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 10 monclient(hunting): picked mon.noname-b con 0x7f86380b3220 addr v1:172.21.15.203:6789/0 2024-03-29T11:43:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 10 monclient(hunting): picked mon.noname-c con 0x7f86381439c0 addr v1:172.21.15.145:6790/0 2024-03-29T11:43:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86381485d0 con 0x7f86380b2800 2024-03-29T11:43:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86381498f0 con 0x7f86380b3220 2024-03-29T11:43:57.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8638149b40 con 0x7f86381439c0 2024-03-29T11:43:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:43:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.017+0000 7f863c8c1640 10 monclient(hunting): authenticate will time out at 1426.324379s 2024-03-29T11:43:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f8637fff640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f86381439c0 0x7f863813b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:41367/0 (socket says 172.21.15.145:41367) 2024-03-29T11:43:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f8637fff640 1 -- 172.21.15.145:0/423439195 learned_addr learned my addr 172.21.15.145:0/423439195 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:43:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f861ffff640 10 client.?.objecter ms_handle_connect 0x7f86381439c0 2024-03-29T11:43:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f861ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:43:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226662503 0 0) 0x7f8638149b40 con 0x7f86381439c0 2024-03-29T11:43:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f861ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:43:57.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f861ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:43:57.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f861ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:43:57.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f861ffff640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:43:57.028 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.021+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8624004eb0 con 0x7f86381439c0 2024-03-29T11:43:57.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.024+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3197920141 0 0) 0x7f8624004eb0 con 0x7f86381439c0 2024-03-29T11:43:57.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.024+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8638149b40 con 0x7f86381439c0 2024-03-29T11:43:57.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.024+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f862c0047e0 con 0x7f86381439c0 2024-03-29T11:43:57.035 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.024+0000 7f861ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:57.035 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.024+0000 7f861ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:43:57.035 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.024+0000 7f861ffff640 10 monclient(hunting): dump: 2024-03-29T11:43:57.035 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:57.035 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:57.035 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:57.035 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:57.035 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:57.036 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:57.036 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:57.036 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:57.036 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:57.036 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:57.036 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.024+0000 7f861ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:43:57.036 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 435121611 0 0) 0x7f8638149b40 con 0x7f86381439c0 2024-03-29T11:43:57.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 >> v1:172.21.15.203:6789/0 conn(0x7f86380b3220 legacy=0x7f86381db0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:57.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 >> v1:172.21.15.145:6789/0 conn(0x7f86380b2800 legacy=0x7f8638147ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:57.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:43:57.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 1 monclient: found mon.c 2024-03-29T11:43:57.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:43:57.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:57.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86381db7c0 con 0x7f86381439c0 2024-03-29T11:43:57.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 10 monclient: _finish_auth 0 2024-03-29T11:43:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:43:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f861ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:43:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f863c8c1640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:43:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f863c8c1640 10 monclient: _renew_subs 2024-03-29T11:43:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f863c8c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863813c040 con 0x7f86381439c0 2024-03-29T11:43:57.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f863c8c1640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:43:57.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f863c8c1640 10 monclient: _renew_subs 2024-03-29T11:43:57.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f863c8c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:57.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f863814a420 con 0x7f86381439c0 2024-03-29T11:43:57.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.025+0000 7f863c8c1640 1 librados: init done 2024-03-29T11:43:57.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f863c8c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:57.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f863c8c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:57.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f863c8c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:57.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863804f730 con 0x7f86381439c0 2024-03-29T11:43:57.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f862c001b10 con 0x7f86381439c0 2024-03-29T11:43:57.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f861ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:43:57.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f862c004d90 con 0x7f86381439c0 2024-03-29T11:43:57.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f861ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:43:57.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f861ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:43:57.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.027+0000 7f861ffff640 10 monclient: dump: 2024-03-29T11:43:57.041 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:43:57.041 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:43:57.041 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:57.041 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:43:57.041 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:43:57.041 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:43:57.041 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:43:57.041 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.028+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f862c004fb0 con 0x7f86381439c0 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.029+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f862c054940 con 0x7f86381439c0 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.029+0000 7f861ffff640 10 client.4577.objecter ms_dispatch 0x7f8638000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.029+0000 7f861ffff640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.029+0000 7f861ffff640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.029+0000 7f861ffff640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.028+0000 7f8635ffb640 4 set_mon_vals no callback set 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.030+0000 7f86377fe640 10 monclient: get_auth_request con 0x7f862406d9e0 auth_method 0 2024-03-29T11:43:57.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.030+0000 7f861ffff640 10 client.4577.objecter ms_handle_connect 0x7f862406d9e0 2024-03-29T11:43:57.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.037+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f862c050be0 con 0x7f86381439c0 2024-03-29T11:43:57.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.037+0000 7f861ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:43:57.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.037+0000 7f861ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:43:57.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.359+0000 7f863c8c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:57.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.359+0000 7f863c8c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:57.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.359+0000 7f863c8c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:43:57.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.359+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8638148770 con 0x7f86381439c0 2024-03-29T11:43:57.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.360+0000 7f861ffff640 1 -- 172.21.15.145:0/423439195 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f862c050d80 con 0x7f86381439c0 2024-03-29T11:43:57.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.360+0000 7f861ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:43:57.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.360+0000 7f861ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:43:57.365 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:43:57.365 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:43:57.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.365+0000 7f863c8c1640 10 librados: watch_flush enter 2024-03-29T11:43:57.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.365+0000 7f863c8c1640 10 librados: watch_flush exit 2024-03-29T11:43:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.365+0000 7f863c8c1640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:43:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.365+0000 7f863c8c1640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:43:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.365+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 >> v1:172.21.15.145:6816/227372239 conn(0x7f862406d9e0 legacy=0x7f862406fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.365+0000 7f863c8c1640 10 monclient: shutdown 2024-03-29T11:43:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.365+0000 7f863c8c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:43:57.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.365+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 >> v1:172.21.15.145:6790/0 conn(0x7f86381439c0 legacy=0x7f863813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:43:57.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.367+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 shutdown_connections 2024-03-29T11:43:57.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.367+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 >> 172.21.15.145:0/423439195 conn(0x7f8638066a00 msgr2=0x7f8638066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:43:57.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.367+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 shutdown_connections 2024-03-29T11:43:57.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.367+0000 7f863c8c1640 1 -- 172.21.15.145:0/423439195 wait complete. 2024-03-29T11:43:57.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:43:57.368+0000 7f863c8c1640 1 librados: shutdown 2024-03-29T11:43:57.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:04.195 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.348+0000 7fe0f4e71640 10 monclient: get_monmap_and_config 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.348+0000 7fe0f4e71640 10 monclient: build_initial_monmap 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.348+0000 7fe0f4e71640 10 monclient: monmap: 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:44:04.349184+0000 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:44:04.349184+0000 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:04.352 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:44:04.353 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:44:04.353 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:04.353 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.350+0000 7fe0f4e71640 1 Processor -- start 2024-03-29T11:44:04.353 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.350+0000 7fe0f4e71640 1 -- start start 2024-03-29T11:44:04.353 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.350+0000 7fe0f4e71640 10 monclient: init 2024-03-29T11:44:04.354 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:04.354 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:04.354 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 10 monclient(hunting): picked mon.noname-a con 0x7fe0e8038c60 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:04.354 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 10 monclient(hunting): picked mon.noname-b con 0x7fe0e80324b0 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:04.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 10 monclient(hunting): picked mon.noname-c con 0x7fe0e80d11e0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:04.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e80caf80 con 0x7fe0e8038c60 2024-03-29T11:44:04.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e8039850 con 0x7fe0e80d11e0 2024-03-29T11:44:04.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e80c78d0 con 0x7fe0e80324b0 2024-03-29T11:44:04.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:04.356 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.351+0000 7fe0f4e71640 10 monclient(hunting): authenticate will time out at 1433.658776s 2024-03-29T11:44:04.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.355+0000 7fe0eeffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fe0e80324b0 0x7fe0e80cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:42345/0 (socket says 172.21.15.145:42345) 2024-03-29T11:44:04.358 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.355+0000 7fe0eeffd640 1 -- 172.21.15.145:0/1155580478 learned_addr learned my addr 172.21.15.145:0/1155580478 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.356+0000 7fe0ee7fc640 1 -- 172.21.15.145:0/1155580478 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724729363 0 0) 0x7fe0e80c78d0 con 0x7fe0e80324b0 2024-03-29T11:44:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.356+0000 7fe0ee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.356+0000 7fe0ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.356+0000 7fe0ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.356+0000 7fe0ee7fc640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:44:04.359 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.356+0000 7fe0ee7fc640 1 -- 172.21.15.145:0/1155580478 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e0003520 con 0x7fe0e80324b0 2024-03-29T11:44:04.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 1 -- 172.21.15.145:0/1155580478 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1830833036 0 0) 0x7fe0e0003520 con 0x7fe0e80324b0 2024-03-29T11:44:04.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 1 -- 172.21.15.145:0/1155580478 >> v1:172.21.15.203:6789/0 conn(0x7fe0e80d11e0 legacy=0x7fe0e80d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:04.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 1 -- 172.21.15.145:0/1155580478 >> v1:172.21.15.145:6789/0 conn(0x7fe0e8038c60 legacy=0x7fe0e80c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:04.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:04.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 1 monclient: found mon.noname-b 2024-03-29T11:44:04.362 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 1 -- 172.21.15.145:0/1155580478 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e80c7a60 con 0x7fe0e80324b0 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 1 -- 172.21.15.145:0/1155580478 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe0dc001690 con 0x7fe0e80324b0 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0ee7fc640 10 monclient: dump: 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:04.363 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:04.364 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:04.364 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:04.364 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:04.364 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:04.364 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:04.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0f4e71640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:44:04.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.358+0000 7fe0f4e71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.359+0000 7fe0ee7fc640 1 -- 172.21.15.145:0/1155580478 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0dc001860 con 0x7fe0e80324b0 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.359+0000 7fe0ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.359+0000 7fe0f4e71640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.359+0000 7fe0f4e71640 4 set_mon_vals no callback set 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.360+0000 7fe0f4e71640 10 monclient: shutdown 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.360+0000 7fe0f4e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.360+0000 7fe0f4e71640 1 -- 172.21.15.145:0/1155580478 >> v1:172.21.15.145:6790/0 conn(0x7fe0e80324b0 legacy=0x7fe0e80cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.360+0000 7fe0ee7fc640 1 -- 172.21.15.145:0/1155580478 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe0dc001cb0 con 0x7fe0e80324b0 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.360+0000 7fe0ee7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.361+0000 7fe0f4e71640 1 -- 172.21.15.145:0/1155580478 shutdown_connections 2024-03-29T11:44:04.365 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.361+0000 7fe0f4e71640 1 -- 172.21.15.145:0/1155580478 >> 172.21.15.145:0/1155580478 conn(0x7fe0e8005c20 msgr2=0x7fe0e8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.362+0000 7fe0f4e71640 1 -- 172.21.15.145:0/1155580478 shutdown_connections 2024-03-29T11:44:04.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.362+0000 7fe0f4e71640 1 -- 172.21.15.145:0/1155580478 wait complete. 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.363+0000 7fe0f4e71640 10 monclient: build_initial_monmap 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.363+0000 7fe0f4e71640 10 monclient: monmap: 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:44:04.370 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.364+0000 7fe0f4e71640 1 librados: starting msgr at 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.364+0000 7fe0f4e71640 1 librados: starting objecter 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.364+0000 7fe0f4e71640 1 Processor -- start 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.364+0000 7fe0f4e71640 1 -- start start 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.364+0000 7fe0f4e71640 1 librados: setting wanted keys 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.364+0000 7fe0f4e71640 1 librados: calling monclient init 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.364+0000 7fe0f4e71640 10 monclient: init 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.365+0000 7fe0f4e71640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.365+0000 7fe0f4e71640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.365+0000 7fe0f4e71640 10 monclient(hunting): picked mon.noname-a con 0x7fe0e80324b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.365+0000 7fe0f4e71640 10 monclient(hunting): picked mon.noname-c con 0x7fe0e8038c60 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.366+0000 7fe0f4e71640 10 monclient(hunting): picked mon.noname-b con 0x7fe0e80d11e0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:04.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.367+0000 7fe0f4e71640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e817e4b0 con 0x7fe0e80324b0 2024-03-29T11:44:04.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.367+0000 7fe0f4e71640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e8034460 con 0x7fe0e80d11e0 2024-03-29T11:44:04.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.367+0000 7fe0f4e71640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0e80346d0 con 0x7fe0e8038c60 2024-03-29T11:44:04.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.367+0000 7fe0f4e71640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:04.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.367+0000 7fe0f4e71640 10 monclient(hunting): authenticate will time out at 1433.674136s 2024-03-29T11:44:04.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.370+0000 7fe0ef7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fe0e80324b0 0x7fe0e80c40f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:59983/0 (socket says 172.21.15.145:59983) 2024-03-29T11:44:04.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.370+0000 7fe0ef7fe640 1 -- 172.21.15.145:0/2452691175 learned_addr learned my addr 172.21.15.145:0/2452691175 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:04.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.370+0000 7fe0dbfff640 10 client.?.objecter ms_handle_connect 0x7fe0e80324b0 2024-03-29T11:44:04.373 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.370+0000 7fe0dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.371+0000 7fe0dbfff640 10 client.?.objecter ms_handle_connect 0x7fe0e80d11e0 2024-03-29T11:44:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.371+0000 7fe0dbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.371+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431203606 0 0) 0x7fe0e817e4b0 con 0x7fe0e80324b0 2024-03-29T11:44:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.371+0000 7fe0dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.371+0000 7fe0dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.371+0000 7fe0dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:04.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.371+0000 7fe0dbfff640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.371+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d4004eb0 con 0x7fe0e80324b0 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.371+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671825311 0 0) 0x7fe0e8034460 con 0x7fe0e80d11e0 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e817e4b0 con 0x7fe0e80d11e0 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3744127490 0 0) 0x7fe0d4004eb0 con 0x7fe0e80324b0 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0e8034460 con 0x7fe0e80324b0 2024-03-29T11:44:04.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe0e4001940 con 0x7fe0e80324b0 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.372+0000 7fe0dbfff640 10 monclient(hunting): dump: 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:04.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2544720178 0 0) 0x7fe0e8034460 con 0x7fe0e80324b0 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 >> v1:172.21.15.145:6790/0 conn(0x7fe0e8038c60 legacy=0x7fe0e817e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 >> v1:172.21.15.203:6789/0 conn(0x7fe0e80d11e0 legacy=0x7fe0e8188ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 1 monclient: found mon.a 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e80c93a0 con 0x7fe0e80324b0 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 10 monclient: _finish_auth 0 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0f4e71640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0f4e71640 10 monclient: _renew_subs 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0f4e71640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:04.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e80c3350 con 0x7fe0e80324b0 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0f4e71640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0f4e71640 10 monclient: _renew_subs 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0f4e71640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e817f050 con 0x7fe0e80324b0 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0f4e71640 1 librados: init done 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0e4001c40 con 0x7fe0e80324b0 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe0e4003a40 con 0x7fe0e80324b0 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.373+0000 7fe0dbfff640 10 monclient: dump: 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:04.382 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:04.383 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:04.383 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:04.383 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:04.383 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:04.383 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:04.383 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.375+0000 7fe0f4e71640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.375+0000 7fe0f4e71640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:04.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.375+0000 7fe0f4e71640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.375+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e8007b90 con 0x7fe0e80324b0 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.377+0000 7fe0edffb640 4 set_mon_vals no callback set 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.378+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fe0e4003ba0 con 0x7fe0e80324b0 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.379+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fe0e4084b40 con 0x7fe0e80324b0 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.379+0000 7fe0dbfff640 10 client.4545.objecter ms_dispatch 0x7fe0f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.380+0000 7fe0dbfff640 3 client.4545.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.380+0000 7fe0dbfff640 3 client.4545.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.380+0000 7fe0dbfff640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.380+0000 7fe0eeffd640 10 monclient: get_auth_request con 0x7fe0d406d9d0 auth_method 0 2024-03-29T11:44:04.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.381+0000 7fe0dbfff640 10 client.4545.objecter ms_handle_connect 0x7fe0d406d9d0 2024-03-29T11:44:04.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.387+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0e4050420 con 0x7fe0e80324b0 2024-03-29T11:44:04.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.387+0000 7fe0dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:04.390 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.387+0000 7fe0dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:04.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.709+0000 7fe0f4e71640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:04.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.709+0000 7fe0f4e71640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:04.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.709+0000 7fe0f4e71640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:04.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.709+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0e8004cd0 con 0x7fe0e80324b0 2024-03-29T11:44:04.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.712+0000 7fe0dbfff640 1 -- 172.21.15.145:0/2452691175 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fe0e4052350 con 0x7fe0e80324b0 2024-03-29T11:44:04.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.712+0000 7fe0dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:04.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.712+0000 7fe0dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:04.716 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:44:04.716 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:04.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.716+0000 7fe0f4e71640 10 librados: watch_flush enter 2024-03-29T11:44:04.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.717+0000 7fe0f4e71640 10 librados: watch_flush exit 2024-03-29T11:44:04.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.717+0000 7fe0f4e71640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:44:04.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.717+0000 7fe0f4e71640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:44:04.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.717+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 >> v1:172.21.15.145:6816/227372239 conn(0x7fe0d406d9d0 legacy=0x7fe0d406fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:04.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.717+0000 7fe0f4e71640 10 monclient: shutdown 2024-03-29T11:44:04.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.717+0000 7fe0f4e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:04.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.717+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 >> v1:172.21.15.145:6789/0 conn(0x7fe0e80324b0 legacy=0x7fe0e80c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:04.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.717+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 shutdown_connections 2024-03-29T11:44:04.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.717+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 >> 172.21.15.145:0/2452691175 conn(0x7fe0e8005c20 msgr2=0x7fe0e8006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:04.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.718+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 shutdown_connections 2024-03-29T11:44:04.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.718+0000 7fe0f4e71640 1 -- 172.21.15.145:0/2452691175 wait complete. 2024-03-29T11:44:04.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:04.718+0000 7fe0f4e71640 1 librados: shutdown 2024-03-29T11:44:04.738 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:11.545 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.674+0000 7f09494a8640 10 monclient: get_monmap_and_config 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.674+0000 7f09494a8640 10 monclient: build_initial_monmap 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.674+0000 7f09494a8640 10 monclient: monmap: 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:44:11.676185+0000 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:44:11.676185+0000 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:11.679 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:44:11.680 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:44:11.680 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:11.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 1 Processor -- start 2024-03-29T11:44:11.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 1 -- start start 2024-03-29T11:44:11.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 10 monclient: init 2024-03-29T11:44:11.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:11.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:11.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 10 monclient(hunting): picked mon.noname-b con 0x7f09441422b0 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 10 monclient(hunting): picked mon.noname-a con 0x7f0944142690 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 10 monclient(hunting): picked mon.noname-c con 0x7f09440b17d0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0944142c80 con 0x7f0944142690 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.676+0000 7f09494a8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09440aae40 con 0x7f09440b17d0 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f09494a8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f094413fbd0 con 0x7f09441422b0 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f09494a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f09494a8640 10 monclient(hunting): authenticate will time out at 1440.985037s 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0942ffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f09441422b0 0x7f094413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:53204/0 (socket says 172.21.15.145:53204) 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0942ffd640 1 -- 172.21.15.145:0/499966373 learned_addr learned my addr 172.21.15.145:0/499966373 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0941ffb640 1 -- 172.21.15.145:0/499966373 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320714832 0 0) 0x7f094413fbd0 con 0x7f09441422b0 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0941ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0941ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:11.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0941ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:11.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0941ffb640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:44:11.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0941ffb640 1 -- 172.21.15.145:0/499966373 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f092c003520 con 0x7f09441422b0 2024-03-29T11:44:11.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0941ffb640 1 -- 172.21.15.145:0/499966373 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3231023769 0 0) 0x7f092c003520 con 0x7f09441422b0 2024-03-29T11:44:11.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0941ffb640 1 -- 172.21.15.145:0/499966373 >> v1:172.21.15.203:6789/0 conn(0x7f09440b17d0 legacy=0x7f09440b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:11.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.677+0000 7f0941ffb640 1 -- 172.21.15.145:0/499966373 >> v1:172.21.15.145:6789/0 conn(0x7f0944142690 legacy=0x7f09440b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:44:11.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:11.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 1 monclient: found mon.noname-b 2024-03-29T11:44:11.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:11.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:44:11.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 1 -- 172.21.15.145:0/499966373 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0944151940 con 0x7f09441422b0 2024-03-29T11:44:11.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:11.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:11.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:11.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 1 -- 172.21.15.145:0/499966373 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0934001690 con 0x7f09441422b0 2024-03-29T11:44:11.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f09494a8640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:44:11.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f09494a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:11.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:11.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:11.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f0941ffb640 10 monclient: dump: 2024-03-29T11:44:11.690 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:11.690 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:11.690 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:11.690 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.679+0000 7f09494a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.680+0000 7f0941ffb640 1 -- 172.21.15.145:0/499966373 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09340018b0 con 0x7f09441422b0 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.680+0000 7f0941ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.680+0000 7f0941ffb640 1 -- 172.21.15.145:0/499966373 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0934001cc0 con 0x7f09441422b0 2024-03-29T11:44:11.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.680+0000 7f0941ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.680+0000 7f0941ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.680+0000 7f0941ffb640 10 monclient: dump: 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:11.692 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.680+0000 7f09494a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.680+0000 7f09494a8640 4 set_mon_vals no callback set 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.681+0000 7f09494a8640 10 monclient: shutdown 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.681+0000 7f09494a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.681+0000 7f09494a8640 1 -- 172.21.15.145:0/499966373 >> v1:172.21.15.145:6790/0 conn(0x7f09441422b0 legacy=0x7f094413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.681+0000 7f09494a8640 1 -- 172.21.15.145:0/499966373 shutdown_connections 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.681+0000 7f09494a8640 1 -- 172.21.15.145:0/499966373 >> 172.21.15.145:0/499966373 conn(0x7f0944066a00 msgr2=0x7f0944066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.681+0000 7f09494a8640 1 -- 172.21.15.145:0/499966373 shutdown_connections 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.681+0000 7f09494a8640 1 -- 172.21.15.145:0/499966373 wait complete. 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.682+0000 7f09494a8640 10 monclient: build_initial_monmap 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.682+0000 7f09494a8640 10 monclient: monmap: 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:11.693 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.682+0000 7f09494a8640 1 librados: starting msgr at 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.682+0000 7f09494a8640 1 librados: starting objecter 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.683+0000 7f09494a8640 1 Processor -- start 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.683+0000 7f09494a8640 1 -- start start 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.683+0000 7f09494a8640 1 librados: setting wanted keys 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.683+0000 7f09494a8640 1 librados: calling monclient init 2024-03-29T11:44:11.694 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.683+0000 7f09494a8640 10 monclient: init 2024-03-29T11:44:11.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.683+0000 7f09494a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:11.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.683+0000 7f09494a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:11.697 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f09494a8640 10 monclient(hunting): picked mon.noname-c con 0x7f09440b17d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f09494a8640 10 monclient(hunting): picked mon.noname-b con 0x7f09441422b0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f09494a8640 10 monclient(hunting): picked mon.noname-a con 0x7f0944142690 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f09494a8640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09440ac850 con 0x7f0944142690 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f09494a8640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09440acac0 con 0x7f09441422b0 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f09494a8640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09441315c0 con 0x7f09440b17d0 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f09494a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f09494a8640 10 monclient(hunting): authenticate will time out at 1440.992093s 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f0942ffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f09440b17d0 0x7f094412fd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:55912/0 (socket says 172.21.15.145:55912) 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f0942ffd640 1 -- 172.21.15.145:0/1292230774 learned_addr learned my addr 172.21.15.145:0/1292230774 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f092b7fe640 10 client.?.objecter ms_handle_connect 0x7f09440b17d0 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f092b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051874042 0 0) 0x7f09441315c0 con 0x7f09440b17d0 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f092b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:11.698 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f092b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f092b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f092b7fe640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.684+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0930004eb0 con 0x7f09440b17d0 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1168282089 0 0) 0x7f0930004eb0 con 0x7f09440b17d0 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09441315c0 con 0x7f09440b17d0 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0934001bc0 con 0x7f09440b17d0 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 monclient(hunting): dump: 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:11.699 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 client.?.objecter ms_handle_connect 0x7f09441422b0 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321510865 0 0) 0x7f09440acac0 con 0x7f09441422b0 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 10 monclient(hunting): my global_id is 4558 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0930004eb0 con 0x7f09441422b0 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3992103849 0 0) 0x7f09441315c0 con 0x7f09440b17d0 2024-03-29T11:44:11.700 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 >> v1:172.21.15.203:6789/0 conn(0x7f09441422b0 legacy=0x7f0944130440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.685+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 >> v1:172.21.15.145:6789/0 conn(0x7f0944142690 legacy=0x7f09441e1c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.686+0000 7f092b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.686+0000 7f092b7fe640 1 monclient: found mon.c 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.686+0000 7f092b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0944131750 con 0x7f09440b17d0 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09494a8640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09494a8640 10 monclient: _renew_subs 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09494a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09494a8640 1 -- 172.21.15.145:0/1292230774 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09440ab540 con 0x7f09440b17d0 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09494a8640 10 client.4586.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:11.701 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09494a8640 10 monclient: _renew_subs 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09494a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09494a8640 1 -- 172.21.15.145:0/1292230774 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0944132b30 con 0x7f09440b17d0 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09494a8640 1 librados: init done 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0934004bf0 con 0x7f09440b17d0 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0934004f40 con 0x7f09440b17d0 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f092b7fe640 10 monclient: dump: 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:11.702 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.687+0000 7f09417fa640 4 set_mon_vals no callback set 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.689+0000 7f09297fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.689+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f09340050a0 con 0x7f09440b17d0 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.689+0000 7f09297fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.689+0000 7f09297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.689+0000 7f09297fa640 1 -- 172.21.15.145:0/1292230774 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0944075f10 con 0x7f09440b17d0 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.690+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f0934084cb0 con 0x7f09440b17d0 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.690+0000 7f09427fc640 10 monclient: get_auth_request con 0x7f093006ddf0 auth_method 0 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.691+0000 7f092b7fe640 10 client.4586.objecter ms_dispatch 0x7f0944000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.691+0000 7f092b7fe640 3 client.4586.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:11.703 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.691+0000 7f092b7fe640 3 client.4586.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:11.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.691+0000 7f092b7fe640 20 client.4586.objecter dump_active .. 0 homeless 2024-03-29T11:44:11.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.691+0000 7f092b7fe640 10 client.4586.objecter ms_handle_connect 0x7f093006ddf0 2024-03-29T11:44:11.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.696+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0934050590 con 0x7f09440b17d0 2024-03-29T11:44:11.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.696+0000 7f092b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:11.704 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:11.696+0000 7f092b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:12.007 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.001+0000 7f09297fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:12.008 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.001+0000 7f09297fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:12.008 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.001+0000 7f09297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:12.008 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.001+0000 7f09297fa640 1 -- 172.21.15.145:0/1292230774 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09440b2270 con 0x7f09440b17d0 2024-03-29T11:44:12.008 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.002+0000 7f092b7fe640 1 -- 172.21.15.145:0/1292230774 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f09340524c0 con 0x7f09440b17d0 2024-03-29T11:44:12.008 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.002+0000 7f092b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:12.008 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.002+0000 7f092b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:12.008 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:44:12.008 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:12.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.005+0000 7f09297fa640 10 librados: watch_flush enter 2024-03-29T11:44:12.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.005+0000 7f09297fa640 10 librados: watch_flush exit 2024-03-29T11:44:12.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.005+0000 7f09297fa640 20 client.4586.objecter shutdown clearing up homeless session... 2024-03-29T11:44:12.009 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.005+0000 7f09297fa640 10 client.4586.objecter successfully canceled tick 2024-03-29T11:44:12.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.006+0000 7f09297fa640 1 -- 172.21.15.145:0/1292230774 >> v1:172.21.15.145:6816/227372239 conn(0x7f093006ddf0 legacy=0x7f09300702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:12.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.006+0000 7f09297fa640 10 monclient: shutdown 2024-03-29T11:44:12.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.006+0000 7f09297fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:12.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.006+0000 7f09297fa640 1 -- 172.21.15.145:0/1292230774 >> v1:172.21.15.145:6790/0 conn(0x7f09440b17d0 legacy=0x7f094412fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:12.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.006+0000 7f09297fa640 1 -- 172.21.15.145:0/1292230774 shutdown_connections 2024-03-29T11:44:12.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.006+0000 7f09297fa640 1 -- 172.21.15.145:0/1292230774 >> 172.21.15.145:0/1292230774 conn(0x7f0944066a00 msgr2=0x7f094413dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:12.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.006+0000 7f09297fa640 1 -- 172.21.15.145:0/1292230774 shutdown_connections 2024-03-29T11:44:12.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.006+0000 7f09297fa640 1 -- 172.21.15.145:0/1292230774 wait complete. 2024-03-29T11:44:12.011 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:12.006+0000 7f09297fa640 1 librados: shutdown 2024-03-29T11:44:12.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:18.832 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:18.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.973+0000 7f1b8359e640 10 monclient: get_monmap_and_config 2024-03-29T11:44:18.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.973+0000 7f1b8359e640 10 monclient: build_initial_monmap 2024-03-29T11:44:18.977 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.973+0000 7f1b8359e640 10 monclient: monmap: 2024-03-29T11:44:18.977 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:18.978 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:18.978 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:44:18.974566+0000 2024-03-29T11:44:18.978 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:44:18.974566+0000 2024-03-29T11:44:18.978 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:18.978 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:18.978 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:18.978 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:44:18.978 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:44:18.978 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:18.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.974+0000 7f1b8359e640 1 Processor -- start 2024-03-29T11:44:18.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.974+0000 7f1b8359e640 1 -- start start 2024-03-29T11:44:18.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.974+0000 7f1b8359e640 10 monclient: init 2024-03-29T11:44:18.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.975+0000 7f1b8359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:18.979 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.975+0000 7f1b8359e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:18.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.975+0000 7f1b8359e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b7c007510 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:18.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.976+0000 7f1b8359e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b7c0078f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:18.980 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.976+0000 7f1b8359e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b7c0c5d40 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:18.981 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.976+0000 7f1b8359e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c02e0b0 con 0x7f1b7c007510 2024-03-29T11:44:18.981 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.976+0000 7f1b8359e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c033840 con 0x7f1b7c0078f0 2024-03-29T11:44:18.981 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.976+0000 7f1b8359e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c02e320 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.981 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.976+0000 7f1b8359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:18.981 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.976+0000 7f1b8359e640 10 monclient(hunting): authenticate will time out at 1448.284473s 2024-03-29T11:44:18.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.979+0000 7f1b8259c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f1b7c007510 0x7f1b7c033130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:37445/0 (socket says 172.21.15.145:37445) 2024-03-29T11:44:18.983 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.979+0000 7f1b8259c640 1 -- 172.21.15.145:0/3726202761 learned_addr learned my addr 172.21.15.145:0/3726202761 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:18.986 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348877202 0 0) 0x7f1b7c02e0b0 con 0x7f1b7c007510 2024-03-29T11:44:18.986 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:18.986 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:18.986 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:18.986 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:44:18.986 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b6c003520 con 0x7f1b7c007510 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498071915 0 0) 0x7f1b7c02e320 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.980+0000 7f1b8159a640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b7c02e0b0 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996282498 0 0) 0x7f1b7c033840 con 0x7f1b7c0078f0 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient(hunting): my global_id is 4561 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b7c02e320 con 0x7f1b7c0078f0 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 261992613 0 0) 0x7f1b7c02e0b0 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.987 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 >> v1:172.21.15.203:6789/0 conn(0x7f1b7c0078f0 legacy=0x7f1b7c0c5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 >> v1:172.21.15.145:6789/0 conn(0x7f1b7c007510 legacy=0x7f1b7c033130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: _finish_hunting 0 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 monclient: found mon.noname-b 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c02e600 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: _finish_auth 0 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: _check_auth_tickets 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1b8404f730 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: dump: 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:18.988 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8359e640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b84075f10 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 1 -- 172.21.15.145:0/3726202761 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1b840758b0 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:18.989 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8159a640 10 monclient: dump: 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.981+0000 7f1b8359e640 4 set_mon_vals no callback set 2024-03-29T11:44:18.990 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.982+0000 7f1b8359e640 10 monclient: shutdown 2024-03-29T11:44:18.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.982+0000 7f1b8359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:18.991 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.982+0000 7f1b8359e640 1 -- 172.21.15.145:0/3726202761 >> v1:172.21.15.145:6790/0 conn(0x7f1b7c0c5d40 legacy=0x7f1b7c0d0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:18.993 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.982+0000 7f1b8359e640 1 -- 172.21.15.145:0/3726202761 shutdown_connections 2024-03-29T11:44:18.993 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.982+0000 7f1b8359e640 1 -- 172.21.15.145:0/3726202761 >> 172.21.15.145:0/3726202761 conn(0x7f1b7c006550 msgr2=0x7f1b7c006960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:18.993 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.983+0000 7f1b8359e640 1 -- 172.21.15.145:0/3726202761 shutdown_connections 2024-03-29T11:44:18.993 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.983+0000 7f1b8359e640 1 -- 172.21.15.145:0/3726202761 wait complete. 2024-03-29T11:44:18.993 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.984+0000 7f1b8359e640 10 monclient: build_initial_monmap 2024-03-29T11:44:18.993 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.984+0000 7f1b8359e640 10 monclient: monmap: 2024-03-29T11:44:18.993 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:18.993 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:18.993 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.984+0000 7f1b8359e640 1 librados: starting msgr at 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.984+0000 7f1b8359e640 1 librados: starting objecter 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.985+0000 7f1b8359e640 1 Processor -- start 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.985+0000 7f1b8359e640 1 -- start start 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.985+0000 7f1b8359e640 1 librados: setting wanted keys 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.985+0000 7f1b8359e640 1 librados: calling monclient init 2024-03-29T11:44:18.994 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.985+0000 7f1b8359e640 10 monclient: init 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 10 monclient(hunting): picked mon.noname-c con 0x7f1b7c007510 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 10 monclient(hunting): picked mon.noname-a con 0x7f1b7c0c5d40 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 10 monclient(hunting): picked mon.noname-b con 0x7f1b7c15be20 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c15c3d0 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c0c06e0 con 0x7f1b7c15be20 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b7c0c09c0 con 0x7f1b7c007510 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:18.995 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.986+0000 7f1b8359e640 10 monclient(hunting): authenticate will time out at 1448.294450s 2024-03-29T11:44:18.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.987+0000 7f1b82d9d640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f1b7c15be20 0x7f1b7c0c17d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:57000/0 (socket says 172.21.15.145:57000) 2024-03-29T11:44:18.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.987+0000 7f1b82d9d640 1 -- 172.21.15.145:0/533811141 learned_addr learned my addr 172.21.15.145:0/533811141 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:18.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.988+0000 7f1b72ffd640 10 client.?.objecter ms_handle_connect 0x7f1b7c15be20 2024-03-29T11:44:18.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.988+0000 7f1b72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:18.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 client.?.objecter ms_handle_connect 0x7f1b7c0c5d40 2024-03-29T11:44:18.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:18.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 client.?.objecter ms_handle_connect 0x7f1b7c007510 2024-03-29T11:44:18.997 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278571904 0 0) 0x7f1b7c15c3d0 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b64004eb0 con 0x7f1b7c0c5d40 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092423355 0 0) 0x7f1b7c0c09c0 con 0x7f1b7c007510 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.990+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b7c15c3d0 con 0x7f1b7c007510 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3548275724 0 0) 0x7f1b7c15c3d0 con 0x7f1b7c007510 2024-03-29T11:44:18.998 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b7c0c09c0 con 0x7f1b7c007510 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1b74001680 con 0x7f1b7c007510 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:18.999 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2738269005 0 0) 0x7f1b64004eb0 con 0x7f1b7c0c5d40 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b640069d0 con 0x7f1b7c0c5d40 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1b78001850 con 0x7f1b7c0c5d40 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:19.000 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1961786211 0 0) 0x7f1b7c0c09c0 con 0x7f1b7c007510 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 >> v1:172.21.15.203:6789/0 conn(0x7f1b7c15be20 legacy=0x7f1b7c0c17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 >> v1:172.21.15.145:6789/0 conn(0x7f1b7c0c5d40 legacy=0x7f1b7c0c10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 monclient: found mon.c 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b7c0c0ca0 con 0x7f1b7c007510 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:19.001 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b8359e640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b8359e640 10 monclient: _renew_subs 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b8359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7c0c1ee0 con 0x7f1b7c007510 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b8359e640 10 client.4592.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b8359e640 10 monclient: _renew_subs 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b8359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7c0c5090 con 0x7f1b7c007510 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b8359e640 1 librados: init done 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b74001bb0 con 0x7f1b7c007510 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.991+0000 7f1b73fff640 4 set_mon_vals no callback set 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.992+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f1b74001680 con 0x7f1b7c007510 2024-03-29T11:44:19.002 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.992+0000 7f1b72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.992+0000 7f1b72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.992+0000 7f1b72ffd640 10 monclient: dump: 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:19.003 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:19.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.994+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f1b7401a650 con 0x7f1b7c007510 2024-03-29T11:44:19.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.994+0000 7f1b81d9b640 10 monclient: get_auth_request con 0x7f1b6406e3f0 auth_method 0 2024-03-29T11:44:19.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.995+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f1b740014a0 con 0x7f1b7c007510 2024-03-29T11:44:19.004 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.995+0000 7f1b72ffd640 10 client.4592.objecter ms_dispatch 0x7f1b84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:19.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.995+0000 7f1b72ffd640 3 client.4592.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:19.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.995+0000 7f1b72ffd640 3 client.4592.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:19.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.995+0000 7f1b72ffd640 20 client.4592.objecter dump_active .. 0 homeless 2024-03-29T11:44:19.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.995+0000 7f1b72ffd640 10 client.4592.objecter ms_handle_connect 0x7f1b6406e3f0 2024-03-29T11:44:19.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.996+0000 7f1b8359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:19.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.996+0000 7f1b8359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:19.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.996+0000 7f1b8359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:19.005 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:18.996+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7c0c2070 con 0x7f1b7c007510 2024-03-29T11:44:19.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.006+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b74050420 con 0x7f1b7c007510 2024-03-29T11:44:19.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.006+0000 7f1b72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:19.010 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.006+0000 7f1b72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:19.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.323+0000 7f1b8359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:19.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.323+0000 7f1b8359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:19.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.323+0000 7f1b8359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:19.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.323+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b7c005120 con 0x7f1b7c007510 2024-03-29T11:44:19.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.324+0000 7f1b72ffd640 1 -- 172.21.15.145:0/533811141 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f1b74050420 con 0x7f1b7c007510 2024-03-29T11:44:19.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.324+0000 7f1b72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:19.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.324+0000 7f1b72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:19.329 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:44:19.329 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:19.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.328+0000 7f1b8359e640 10 librados: watch_flush enter 2024-03-29T11:44:19.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.328+0000 7f1b8359e640 10 librados: watch_flush exit 2024-03-29T11:44:19.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.328+0000 7f1b8359e640 20 client.4592.objecter shutdown clearing up homeless session... 2024-03-29T11:44:19.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.328+0000 7f1b8359e640 10 client.4592.objecter successfully canceled tick 2024-03-29T11:44:19.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.328+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 >> v1:172.21.15.145:6816/227372239 conn(0x7f1b6406e3f0 legacy=0x7f1b640708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:19.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.328+0000 7f1b8359e640 10 monclient: shutdown 2024-03-29T11:44:19.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.328+0000 7f1b8359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:19.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.328+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 >> v1:172.21.15.145:6790/0 conn(0x7f1b7c007510 legacy=0x7f1b7c0bc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:19.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.329+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 shutdown_connections 2024-03-29T11:44:19.333 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.329+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 >> 172.21.15.145:0/533811141 conn(0x7f1b7c006550 msgr2=0x7f1b7c032e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:19.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.329+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 shutdown_connections 2024-03-29T11:44:19.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.329+0000 7f1b8359e640 1 -- 172.21.15.145:0/533811141 wait complete. 2024-03-29T11:44:19.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:19.332+0000 7f1b8359e640 1 librados: shutdown 2024-03-29T11:44:19.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:26.162 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:26.317 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.310+0000 7fec326db640 10 monclient: get_monmap_and_config 2024-03-29T11:44:26.317 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.310+0000 7fec326db640 10 monclient: build_initial_monmap 2024-03-29T11:44:26.317 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.310+0000 7fec326db640 10 monclient: monmap: 2024-03-29T11:44:26.317 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:26.317 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:26.317 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:44:26.311765+0000 2024-03-29T11:44:26.317 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:44:26.311765+0000 2024-03-29T11:44:26.317 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:26.317 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:26.318 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:26.318 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:44:26.318 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:44:26.318 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:26.318 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.312+0000 7fec326db640 1 Processor -- start 2024-03-29T11:44:26.318 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 1 -- start start 2024-03-29T11:44:26.318 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 10 monclient: init 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 10 monclient(hunting): picked mon.noname-a con 0x7fec2c0ae220 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 10 monclient(hunting): picked mon.noname-c con 0x7fec2c072f10 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 10 monclient(hunting): picked mon.noname-b con 0x7fec2c0acea0 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c0aa9f0 con 0x7fec2c0ae220 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c0ac7a0 con 0x7fec2c072f10 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c0ab5a0 con 0x7fec2c0acea0 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.313+0000 7fec326db640 10 monclient(hunting): authenticate will time out at 1455.622006s 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.317+0000 7fec31eda640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7fec2c0acea0 0x7fec2c0ad2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:37957/0 (socket says 172.21.15.145:37957) 2024-03-29T11:44:26.322 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.317+0000 7fec31eda640 1 -- 172.21.15.145:0/638977194 learned_addr learned my addr 172.21.15.145:0/638977194 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3783427971 0 0) 0x7fec2c0ab5a0 con 0x7fec2c0acea0 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec18003520 con 0x7fec2c0acea0 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137656189 0 0) 0x7fec2c0aa9f0 con 0x7fec2c0ae220 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:26.328 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c0ab5a0 con 0x7fec2c0ae220 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906010944 0 0) 0x7fec2c0ac7a0 con 0x7fec2c072f10 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): _negotiate 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient(hunting): my global_id is 4564 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c0aa9f0 con 0x7fec2c072f10 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 708676775 0 0) 0x7fec18003520 con 0x7fec2c0acea0 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 >> v1:172.21.15.203:6789/0 conn(0x7fec2c072f10 legacy=0x7fec2c141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 >> v1:172.21.15.145:6789/0 conn(0x7fec2c0ae220 legacy=0x7fec2c141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 10 monclient: _finish_hunting 0 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 1 monclient: found mon.noname-b 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.321+0000 7fec23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:26.329 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c0ab730 con 0x7fec2c0acea0 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: _finish_auth 0 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: _check_auth_tickets 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec326db640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec326db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fec1c001690 con 0x7fec2c0acea0 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: dump: 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:26.330 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec326db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec1c001870 con 0x7fec2c0acea0 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 1 -- 172.21.15.145:0/638977194 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fec1c001c00 con 0x7fec2c0acea0 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec23fff640 10 monclient: dump: 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:26.331 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec326db640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.322+0000 7fec326db640 4 set_mon_vals no callback set 2024-03-29T11:44:26.332 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.323+0000 7fec326db640 10 monclient: shutdown 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.323+0000 7fec326db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.323+0000 7fec326db640 1 -- 172.21.15.145:0/638977194 >> v1:172.21.15.145:6790/0 conn(0x7fec2c0acea0 legacy=0x7fec2c0ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.326+0000 7fec326db640 1 -- 172.21.15.145:0/638977194 shutdown_connections 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.326+0000 7fec326db640 1 -- 172.21.15.145:0/638977194 >> 172.21.15.145:0/638977194 conn(0x7fec2c0b3a30 msgr2=0x7fec2c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.326+0000 7fec326db640 1 -- 172.21.15.145:0/638977194 shutdown_connections 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.326+0000 7fec326db640 1 -- 172.21.15.145:0/638977194 wait complete. 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.327+0000 7fec326db640 10 monclient: build_initial_monmap 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.327+0000 7fec326db640 10 monclient: monmap: 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:26.334 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.328+0000 7fec326db640 1 librados: starting msgr at 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.328+0000 7fec326db640 1 librados: starting objecter 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.328+0000 7fec326db640 1 Processor -- start 2024-03-29T11:44:26.335 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.328+0000 7fec326db640 1 -- start start 2024-03-29T11:44:26.339 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.328+0000 7fec326db640 1 librados: setting wanted keys 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.328+0000 7fec326db640 1 librados: calling monclient init 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.328+0000 7fec326db640 10 monclient: init 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 10 monclient(hunting): picked mon.noname-b con 0x7fec2c072f10 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 10 monclient(hunting): picked mon.noname-c con 0x7fec2c0acea0 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 10 monclient(hunting): picked mon.noname-a con 0x7fec2c0ae220 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c13c460 con 0x7fec2c0ae220 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c1324a0 con 0x7fec2c072f10 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec2c13c6b0 con 0x7fec2c0acea0 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.329+0000 7fec326db640 10 monclient(hunting): authenticate will time out at 1455.637585s 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.333+0000 7fec31eda640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fec2c0ae220 0x7fec2c1e6060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:53763/0 (socket says 172.21.15.145:53763) 2024-03-29T11:44:26.340 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.333+0000 7fec31eda640 1 -- 172.21.15.145:0/30099825 learned_addr learned my addr 172.21.15.145:0/30099825 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 10 client.?.objecter ms_handle_connect 0x7fec2c0ae220 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070269368 0 0) 0x7fec2c13c460 con 0x7fec2c0ae220 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec10004eb0 con 0x7fec2c0ae220 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1769058156 0 0) 0x7fec10004eb0 con 0x7fec2c0ae220 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec10005300 con 0x7fec2c0ae220 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fec1c001bc0 con 0x7fec2c0ae220 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.334+0000 7fec21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.335+0000 7fec21ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:26.341 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.335+0000 7fec21ffb640 10 monclient(hunting): dump: 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.335+0000 7fec21ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.335+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1376366030 0 0) 0x7fec10005300 con 0x7fec2c0ae220 2024-03-29T11:44:26.342 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.335+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 >> v1:172.21.15.145:6790/0 conn(0x7fec2c0acea0 legacy=0x7fec2c13d9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:26.345 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 >> v1:172.21.15.203:6789/0 conn(0x7fec2c072f10 legacy=0x7fec2c13bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:26.345 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec21ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:44:26.345 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec21ffb640 1 monclient: found mon.a 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec21ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c13e0d0 con 0x7fec2c0ae220 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec21ffb640 10 monclient: _finish_auth 0 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec21ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec326db640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec326db640 10 monclient: _renew_subs 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec326db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.336+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c13cbd0 con 0x7fec2c0ae220 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.337+0000 7fec326db640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.337+0000 7fec326db640 10 monclient: _renew_subs 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.337+0000 7fec326db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:26.346 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.337+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c140e10 con 0x7fec2c0ae220 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.337+0000 7fec326db640 1 librados: init done 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.339+0000 7fec326db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.339+0000 7fec326db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.339+0000 7fec326db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.339+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c13c840 con 0x7fec2c0ae220 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.340+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec1c004a90 con 0x7fec2c0ae220 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.340+0000 7fec21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.340+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fec1c001bc0 con 0x7fec2c0ae220 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.340+0000 7fec21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.340+0000 7fec21ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.340+0000 7fec21ffb640 10 monclient: dump: 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:26.347 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:26.348 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:26.348 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:26.348 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:26.348 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:26.348 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:26.348 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:26.348 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:26.349 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.341+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fec1c004d40 con 0x7fec2c0ae220 2024-03-29T11:44:26.349 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.342+0000 7fec237fe640 4 set_mon_vals no callback set 2024-03-29T11:44:26.349 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.342+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fec1c0545e0 con 0x7fec2c0ae220 2024-03-29T11:44:26.349 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.342+0000 7fec21ffb640 10 client.4557.objecter ms_dispatch 0x7fec2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:26.349 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.342+0000 7fec21ffb640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:26.349 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.342+0000 7fec21ffb640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:26.349 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.342+0000 7fec316d9640 10 monclient: get_auth_request con 0x7fec1006e1d0 auth_method 0 2024-03-29T11:44:26.349 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.342+0000 7fec21ffb640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-29T11:44:26.350 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.343+0000 7fec21ffb640 10 client.4557.objecter ms_handle_connect 0x7fec1006e1d0 2024-03-29T11:44:26.354 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.350+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec1c050880 con 0x7fec2c0ae220 2024-03-29T11:44:26.354 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.350+0000 7fec21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:26.355 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.350+0000 7fec21ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:26.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.672+0000 7fec326db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:26.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.672+0000 7fec326db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:26.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.672+0000 7fec326db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:26.676 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.672+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec2c04f730 con 0x7fec2c0ae220 2024-03-29T11:44:26.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.673+0000 7fec21ffb640 1 -- 172.21.15.145:0/30099825 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fec1c050880 con 0x7fec2c0ae220 2024-03-29T11:44:26.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.673+0000 7fec21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:26.678 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.673+0000 7fec21ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:26.678 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:44:26.678 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:26.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 10 librados: watch_flush enter 2024-03-29T11:44:26.681 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 10 librados: watch_flush exit 2024-03-29T11:44:26.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-29T11:44:26.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 10 client.4557.objecter successfully canceled tick 2024-03-29T11:44:26.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 >> v1:172.21.15.145:6816/227372239 conn(0x7fec1006e1d0 legacy=0x7fec10070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:26.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 10 monclient: shutdown 2024-03-29T11:44:26.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:26.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 >> v1:172.21.15.145:6789/0 conn(0x7fec2c0ae220 legacy=0x7fec2c1e6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:26.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 shutdown_connections 2024-03-29T11:44:26.682 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.677+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 >> 172.21.15.145:0/30099825 conn(0x7fec2c0b3a30 msgr2=0x7fec2c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:26.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.678+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 shutdown_connections 2024-03-29T11:44:26.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.678+0000 7fec326db640 1 -- 172.21.15.145:0/30099825 wait complete. 2024-03-29T11:44:26.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:26.678+0000 7fec326db640 1 librados: shutdown 2024-03-29T11:44:26.701 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:33.507 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:33.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.654+0000 7f5b42dde640 10 monclient: get_monmap_and_config 2024-03-29T11:44:33.658 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.654+0000 7f5b42dde640 10 monclient: build_initial_monmap 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.654+0000 7f5b42dde640 10 monclient: monmap: 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:44:33.655423+0000 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:44:33.655423+0000 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:44:33.659 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:33.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.655+0000 7f5b42dde640 1 Processor -- start 2024-03-29T11:44:33.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.656+0000 7f5b42dde640 1 -- start start 2024-03-29T11:44:33.660 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.656+0000 7f5b42dde640 10 monclient: init 2024-03-29T11:44:33.661 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.657+0000 7f5b42dde640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:33.662 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.658+0000 7f5b42dde640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:44:33.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.658+0000 7f5b42dde640 10 monclient(hunting): picked mon.noname-b con 0x7f5b340389e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:33.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.659+0000 7f5b42dde640 10 monclient(hunting): picked mon.noname-c con 0x7f5b34038dc0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:33.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.659+0000 7f5b425dd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5b34030660 0x7f5b340d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:44118/0 (socket says 172.21.15.145:44118) 2024-03-29T11:44:33.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.659+0000 7f5b425dd640 1 -- 172.21.15.145:0/1480764141 learned_addr learned my addr 172.21.15.145:0/1480764141 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:33.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.659+0000 7f5b42dde640 10 monclient(hunting): picked mon.noname-a con 0x7f5b34030660 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:33.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.659+0000 7f5b42dde640 1 -- 172.21.15.145:0/1480764141 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b340c54c0 con 0x7f5b34030660 2024-03-29T11:44:33.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.659+0000 7f5b42dde640 1 -- 172.21.15.145:0/1480764141 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b34036430 con 0x7f5b34038dc0 2024-03-29T11:44:33.664 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.659+0000 7f5b42dde640 1 -- 172.21.15.145:0/1480764141 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b34039840 con 0x7f5b340389e0 2024-03-29T11:44:33.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.659+0000 7f5b42dde640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:33.665 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.659+0000 7f5b42dde640 10 monclient(hunting): authenticate will time out at 1462.967448s 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.662+0000 7f5b40dda640 1 -- 172.21.15.145:0/1480764141 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4121870382 0 0) 0x7f5b340c54c0 con 0x7f5b34030660 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient(hunting): _negotiate 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 1 -- 172.21.15.145:0/1480764141 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b28003520 con 0x7f5b34030660 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 1 -- 172.21.15.145:0/1480764141 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3594383411 0 0) 0x7f5b28003520 con 0x7f5b34030660 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 1 -- 172.21.15.145:0/1480764141 >> v1:172.21.15.145:6790/0 conn(0x7f5b340389e0 legacy=0x7f5b34033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 1 -- 172.21.15.145:0/1480764141 >> v1:172.21.15.203:6789/0 conn(0x7f5b34038dc0 legacy=0x7f5b340cadd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: _finish_hunting 0 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 1 monclient: found mon.noname-a 2024-03-29T11:44:33.671 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 1 -- 172.21.15.145:0/1480764141 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b34035690 con 0x7f5b34030660 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: _finish_auth 0 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: _check_auth_tickets 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 1 -- 172.21.15.145:0/1480764141 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5b3c04f730 con 0x7f5b34030660 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: dump: 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:33.672 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b42dde640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b42dde640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 1 -- 172.21.15.145:0/1480764141 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b3c075f10 con 0x7f5b34030660 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 1 -- 172.21.15.145:0/1480764141 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5b3c0758b0 con 0x7f5b34030660 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b40dda640 10 monclient: dump: 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:33.673 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.674 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.674 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:33.674 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:33.674 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:33.674 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:33.674 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:33.675 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:33.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b42dde640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:33.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.663+0000 7f5b42dde640 4 set_mon_vals no callback set 2024-03-29T11:44:33.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.664+0000 7f5b42dde640 10 monclient: shutdown 2024-03-29T11:44:33.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.664+0000 7f5b42dde640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:33.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.664+0000 7f5b42dde640 1 -- 172.21.15.145:0/1480764141 >> v1:172.21.15.145:6789/0 conn(0x7f5b34030660 legacy=0x7f5b340d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:33.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.666+0000 7f5b42dde640 1 -- 172.21.15.145:0/1480764141 shutdown_connections 2024-03-29T11:44:33.675 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.666+0000 7f5b42dde640 1 -- 172.21.15.145:0/1480764141 >> 172.21.15.145:0/1480764141 conn(0x7f5b34005c00 msgr2=0x7f5b34006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:33.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.668+0000 7f5b42dde640 1 -- 172.21.15.145:0/1480764141 shutdown_connections 2024-03-29T11:44:33.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.668+0000 7f5b42dde640 1 -- 172.21.15.145:0/1480764141 wait complete. 2024-03-29T11:44:33.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.669+0000 7f5b42dde640 10 monclient: build_initial_monmap 2024-03-29T11:44:33.683 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.669+0000 7f5b42dde640 10 monclient: monmap: 2024-03-29T11:44:33.683 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:33.683 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:33.683 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:44:33.683 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.669+0000 7f5b42dde640 1 librados: starting msgr at 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.669+0000 7f5b42dde640 1 librados: starting objecter 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.669+0000 7f5b42dde640 1 Processor -- start 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.669+0000 7f5b42dde640 1 -- start start 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.669+0000 7f5b42dde640 1 librados: setting wanted keys 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.669+0000 7f5b42dde640 1 librados: calling monclient init 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.669+0000 7f5b42dde640 10 monclient: init 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:33.684 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:33.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 10 monclient(hunting): picked mon.noname-a con 0x7f5b34030660 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:33.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 10 monclient(hunting): picked mon.noname-c con 0x7f5b340389e0 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:33.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 10 monclient(hunting): picked mon.noname-b con 0x7f5b3402d3c0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:33.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3415ff40 con 0x7f5b34030660 2024-03-29T11:44:33.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3402e9e0 con 0x7f5b3402d3c0 2024-03-29T11:44:33.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b34160ae0 con 0x7f5b340389e0 2024-03-29T11:44:33.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:33.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.670+0000 7f5b42dde640 10 monclient(hunting): authenticate will time out at 1462.978911s 2024-03-29T11:44:33.685 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.672+0000 7f5b41ddc640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5b34030660 0x7f5b340c7b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:45372/0 (socket says 172.21.15.145:45372) 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.672+0000 7f5b41ddc640 1 -- 172.21.15.145:0/2789803290 learned_addr learned my addr 172.21.15.145:0/2789803290 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.672+0000 7f5b327fc640 10 client.?.objecter ms_handle_connect 0x7f5b34030660 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.672+0000 7f5b327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.673+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616513486 0 0) 0x7f5b3415ff40 con 0x7f5b34030660 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.673+0000 7f5b327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.673+0000 7f5b327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.673+0000 7f5b327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.673+0000 7f5b327fc640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.673+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b20004eb0 con 0x7f5b34030660 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 client.?.objecter ms_handle_connect 0x7f5b340389e0 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1654173190 0 0) 0x7f5b20004eb0 con 0x7f5b34030660 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b20005300 con 0x7f5b34030660 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5b2c001920 con 0x7f5b34030660 2024-03-29T11:44:33.686 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): dump: 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:33.687 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833820522 0 0) 0x7f5b34160ae0 con 0x7f5b340389e0 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b20004eb0 con 0x7f5b340389e0 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4027683314 0 0) 0x7f5b20004eb0 con 0x7f5b340389e0 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b20005b60 con 0x7f5b340389e0 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5b380017d0 con 0x7f5b340389e0 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:33.688 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.674+0000 7f5b327fc640 10 monclient(hunting): dump: 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:33.689 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 991124334 0 0) 0x7f5b20005b60 con 0x7f5b340389e0 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 >> v1:172.21.15.203:6789/0 conn(0x7f5b3402d3c0 legacy=0x7f5b3402f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 >> v1:172.21.15.145:6789/0 conn(0x7f5b34030660 legacy=0x7f5b340c7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 1 monclient: found mon.c 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3402feb0 con 0x7f5b340389e0 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b42dde640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:44:33.690 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b42dde640 10 monclient: _renew_subs 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b42dde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b34030100 con 0x7f5b340389e0 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b42dde640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b42dde640 10 monclient: _renew_subs 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b42dde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3402f050 con 0x7f5b340389e0 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.675+0000 7f5b42dde640 1 librados: init done 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b38001c80 con 0x7f5b340389e0 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b42dde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b42dde640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:33.691 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b42dde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b33fff640 4 set_mon_vals no callback set 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b34007b90 con 0x7f5b340389e0 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5b380017d0 con 0x7f5b340389e0 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.677+0000 7f5b327fc640 10 monclient: dump: 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:33.692 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:33.693 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:33.693 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:33.693 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.679+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f5b38019500 con 0x7f5b340389e0 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.679+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f5b380015f0 con 0x7f5b340389e0 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.679+0000 7f5b327fc640 10 client.4598.objecter ms_dispatch 0x7f5b3c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.679+0000 7f5b327fc640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.679+0000 7f5b327fc640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.679+0000 7f5b327fc640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.682+0000 7f5b41ddc640 10 monclient: get_auth_request con 0x7f5b2006e6a0 auth_method 0 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.682+0000 7f5b327fc640 10 client.4598.objecter ms_handle_connect 0x7f5b2006e6a0 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.690+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b3804f320 con 0x7f5b340389e0 2024-03-29T11:44:33.695 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.690+0000 7f5b327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:33.696 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:33.690+0000 7f5b327fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:34.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.022+0000 7f5b42dde640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:34.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.022+0000 7f5b42dde640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:34.026 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.022+0000 7f5b42dde640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:34.027 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.022+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b3415fca0 con 0x7f5b340389e0 2024-03-29T11:44:34.036 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.032+0000 7f5b327fc640 1 -- 172.21.15.145:0/2789803290 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f5b3804f320 con 0x7f5b340389e0 2024-03-29T11:44:34.036 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.032+0000 7f5b327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:34.036 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.032+0000 7f5b327fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:34.039 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:44:34.039 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:34.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.035+0000 7f5b42dde640 10 librados: watch_flush enter 2024-03-29T11:44:34.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 10 librados: watch_flush exit 2024-03-29T11:44:34.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:44:34.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:44:34.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 >> v1:172.21.15.145:6816/227372239 conn(0x7f5b2006e6a0 legacy=0x7f5b20070b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:34.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 10 monclient: shutdown 2024-03-29T11:44:34.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:34.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 >> v1:172.21.15.145:6790/0 conn(0x7f5b340389e0 legacy=0x7f5b341603d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:34.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 shutdown_connections 2024-03-29T11:44:34.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 >> 172.21.15.145:0/2789803290 conn(0x7f5b34005c00 msgr2=0x7f5b34034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:34.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 shutdown_connections 2024-03-29T11:44:34.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.036+0000 7f5b42dde640 1 -- 172.21.15.145:0/2789803290 wait complete. 2024-03-29T11:44:34.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:34.037+0000 7f5b42dde640 1 librados: shutdown 2024-03-29T11:44:34.057 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:40.866 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.016+0000 7ff90d21e640 10 monclient: get_monmap_and_config 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.016+0000 7ff90d21e640 10 monclient: build_initial_monmap 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.016+0000 7ff90d21e640 10 monclient: monmap: 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:44:41.017891+0000 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:44:41.017891+0000 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:44:41.021 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:44:41.022 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.017+0000 7ff90d21e640 1 Processor -- start 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.017+0000 7ff90d21e640 1 -- start start 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.017+0000 7ff90d21e640 10 monclient: init 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 10 monclient(hunting): picked mon.noname-c con 0x7ff9080b0860 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 10 monclient(hunting): picked mon.noname-b con 0x7ff90813ef30 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 10 monclient(hunting): picked mon.noname-a con 0x7ff9080af8e0 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9080b0c40 con 0x7ff9080af8e0 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9080aed10 con 0x7ff9080b0860 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9080afd50 con 0x7ff90813ef30 2024-03-29T11:44:41.029 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.018+0000 7ff90d21e640 10 monclient(hunting): authenticate will time out at 1470.326592s 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff90759e640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff9080af8e0 0x7ff908149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:49172/0 (socket says 172.21.15.145:49172) 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff90759e640 1 -- 172.21.15.145:0/1593895148 learned_addr learned my addr 172.21.15.145:0/1593895148 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291141632 0 0) 0x7ff9080aed10 con 0x7ff9080b0860 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 10 monclient(hunting): my global_id is 4567 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f0003520 con 0x7ff9080b0860 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070489386 0 0) 0x7ff9080b0c40 con 0x7ff9080af8e0 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:41.030 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9080aed10 con 0x7ff9080af8e0 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.021+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289993055 0 0) 0x7ff8f0003520 con 0x7ff9080b0860 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 >> v1:172.21.15.145:6790/0 conn(0x7ff90813ef30 legacy=0x7ff9080af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 >> v1:172.21.15.145:6789/0 conn(0x7ff9080af8e0 legacy=0x7ff908149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 1 monclient: found mon.noname-c 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9080ab1b0 con 0x7ff9080b0860 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: _finish_auth 0 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:41.031 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff900001940 con 0x7ff9080b0860 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: dump: 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff90d21e640 5 monclient: authenticate success, global_id 4567 2024-03-29T11:44:41.032 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff90d21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff900001aa0 con 0x7ff9080b0860 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 1 -- 172.21.15.145:0/1593895148 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff900001eb0 con 0x7ff9080b0860 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff905d9b640 10 monclient: dump: 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:41.033 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff90d21e640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.022+0000 7ff90d21e640 4 set_mon_vals no callback set 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.023+0000 7ff90d21e640 10 monclient: shutdown 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.023+0000 7ff90d21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.023+0000 7ff90d21e640 1 -- 172.21.15.145:0/1593895148 >> v1:172.21.15.203:6789/0 conn(0x7ff9080b0860 legacy=0x7ff90813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.023+0000 7ff90d21e640 1 -- 172.21.15.145:0/1593895148 shutdown_connections 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.023+0000 7ff90d21e640 1 -- 172.21.15.145:0/1593895148 >> 172.21.15.145:0/1593895148 conn(0x7ff90806b660 msgr2=0x7ff90806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.023+0000 7ff90d21e640 1 -- 172.21.15.145:0/1593895148 shutdown_connections 2024-03-29T11:44:41.034 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.023+0000 7ff90d21e640 1 -- 172.21.15.145:0/1593895148 wait complete. 2024-03-29T11:44:41.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.024+0000 7ff90d21e640 10 monclient: build_initial_monmap 2024-03-29T11:44:41.037 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.024+0000 7ff90d21e640 10 monclient: monmap: 2024-03-29T11:44:41.037 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:41.037 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:41.037 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:44:41.037 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:44:41.037 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:41.037 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.025+0000 7ff90d21e640 1 librados: starting msgr at 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.025+0000 7ff90d21e640 1 librados: starting objecter 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.025+0000 7ff90d21e640 1 Processor -- start 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.025+0000 7ff90d21e640 1 -- start start 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.025+0000 7ff90d21e640 1 librados: setting wanted keys 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.025+0000 7ff90d21e640 1 librados: calling monclient init 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.025+0000 7ff90d21e640 10 monclient: init 2024-03-29T11:44:41.038 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 10 monclient(hunting): picked mon.noname-b con 0x7ff9080af8e0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 10 monclient(hunting): picked mon.noname-a con 0x7ff9080b0860 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 10 monclient(hunting): picked mon.noname-c con 0x7ff90813ef30 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff908136f50 con 0x7ff9080b0860 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff908135220 con 0x7ff9080af8e0 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9080b12e0 con 0x7ff90813ef30 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.026+0000 7ff90d21e640 10 monclient(hunting): authenticate will time out at 1470.334568s 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.027+0000 7ff90659c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff9080b0860 0x7ff908137100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:50956/0 (socket says 172.21.15.145:50956) 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.027+0000 7ff90659c640 1 -- 172.21.15.145:0/4003298191 learned_addr learned my addr 172.21.15.145:0/4003298191 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.028+0000 7ff8ef7fe640 10 client.?.objecter ms_handle_connect 0x7ff9080b0860 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.028+0000 7ff8ef7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:41.039 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262995128 0 0) 0x7ff908136f50 con 0x7ff9080b0860 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f4004eb0 con 0x7ff9080b0860 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3306775871 0 0) 0x7ff8f4004eb0 con 0x7ff9080b0860 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff908136f50 con 0x7ff9080b0860 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff8f8001930 con 0x7ff9080b0860 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 10 monclient(hunting): dump: 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:41.040 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.029+0000 7ff8ef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.030+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 863738118 0 0) 0x7ff908136f50 con 0x7ff9080b0860 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.030+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 >> v1:172.21.15.145:6790/0 conn(0x7ff90813ef30 legacy=0x7ff9081eee70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.030+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 >> v1:172.21.15.203:6789/0 conn(0x7ff9080af8e0 legacy=0x7ff908136840 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.030+0000 7ff8ef7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.030+0000 7ff8ef7fe640 1 monclient: found mon.a 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff8ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:41.041 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff8ef7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9080b1550 con 0x7ff9080b0860 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff8ef7fe640 10 monclient: _finish_auth 0 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff8ef7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff8ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff90d21e640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff90d21e640 10 monclient: _renew_subs 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff90d21e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9081c9440 con 0x7ff9080b0860 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff90d21e640 10 client.4569.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:41.042 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff90d21e640 10 monclient: _renew_subs 2024-03-29T11:44:41.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff90d21e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:41.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff90808bd50 con 0x7ff9080b0860 2024-03-29T11:44:41.043 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.031+0000 7ff90d21e640 1 librados: init done 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.033+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8f8003610 con 0x7ff9080b0860 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.033+0000 7ff8ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.033+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7ff8f8003a60 con 0x7ff9080b0860 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.033+0000 7ff8ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.033+0000 7ff8ef7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.033+0000 7ff8ef7fe640 10 monclient: dump: 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:41.045 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.034+0000 7ff904d99640 4 set_mon_vals no callback set 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.035+0000 7ff90d21e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.035+0000 7ff90d21e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.035+0000 7ff90d21e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.035+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e8005080 con 0x7ff9080b0860 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.036+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7ff8f8003bc0 con 0x7ff9080b0860 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.037+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7ff8f8084ab0 con 0x7ff9080b0860 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.037+0000 7ff8ef7fe640 10 client.4569.objecter ms_dispatch 0x7ff908000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:41.046 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.037+0000 7ff8ef7fe640 3 client.4569.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:41.047 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.037+0000 7ff8ef7fe640 3 client.4569.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:41.047 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.037+0000 7ff8ef7fe640 20 client.4569.objecter dump_active .. 0 homeless 2024-03-29T11:44:41.047 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.037+0000 7ff906d9d640 10 monclient: get_auth_request con 0x7ff8f406dc70 auth_method 0 2024-03-29T11:44:41.047 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.038+0000 7ff8ef7fe640 10 client.4569.objecter ms_handle_connect 0x7ff8f406dc70 2024-03-29T11:44:41.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.044+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8f8050390 con 0x7ff9080b0860 2024-03-29T11:44:41.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.044+0000 7ff8ef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:41.048 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.044+0000 7ff8ef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:41.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.366+0000 7ff90d21e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:41.370 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.366+0000 7ff90d21e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:41.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.366+0000 7ff90d21e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:44:41.371 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.366+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8e8005330 con 0x7ff9080b0860 2024-03-29T11:44:41.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.368+0000 7ff8ef7fe640 1 -- 172.21.15.145:0/4003298191 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7ff8f80522c0 con 0x7ff9080b0860 2024-03-29T11:44:41.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.368+0000 7ff8ef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:41.372 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.368+0000 7ff8ef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:41.373 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:44:41.373 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:41.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.371+0000 7ff90d21e640 10 librados: watch_flush enter 2024-03-29T11:44:41.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.372+0000 7ff90d21e640 10 librados: watch_flush exit 2024-03-29T11:44:41.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.372+0000 7ff90d21e640 20 client.4569.objecter shutdown clearing up homeless session... 2024-03-29T11:44:41.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.372+0000 7ff90d21e640 10 client.4569.objecter successfully canceled tick 2024-03-29T11:44:41.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.372+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 >> v1:172.21.15.145:6816/227372239 conn(0x7ff8f406dc70 legacy=0x7ff8f4070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:41.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.372+0000 7ff90d21e640 10 monclient: shutdown 2024-03-29T11:44:41.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.372+0000 7ff90d21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:41.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.372+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 >> v1:172.21.15.145:6789/0 conn(0x7ff9080b0860 legacy=0x7ff908137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:41.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.373+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 shutdown_connections 2024-03-29T11:44:41.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.373+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 >> 172.21.15.145:0/4003298191 conn(0x7ff90806b660 msgr2=0x7ff90813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:41.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.374+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 shutdown_connections 2024-03-29T11:44:41.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.374+0000 7ff90d21e640 1 -- 172.21.15.145:0/4003298191 wait complete. 2024-03-29T11:44:41.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:41.374+0000 7ff90d21e640 1 librados: shutdown 2024-03-29T11:44:41.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:48.203 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:48.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.358+0000 7f445cc40640 10 monclient: get_monmap_and_config 2024-03-29T11:44:48.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.358+0000 7f445cc40640 10 monclient: build_initial_monmap 2024-03-29T11:44:48.361 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.358+0000 7f445cc40640 10 monclient: monmap: 2024-03-29T11:44:48.361 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:48.362 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:48.362 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:44:48.358425+0000 2024-03-29T11:44:48.362 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:44:48.358425+0000 2024-03-29T11:44:48.362 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:48.362 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:48.362 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:48.362 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:44:48.362 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:44:48.362 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:48.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.358+0000 7f445cc40640 1 Processor -- start 2024-03-29T11:44:48.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.359+0000 7f445cc40640 1 -- start start 2024-03-29T11:44:48.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.359+0000 7f445cc40640 10 monclient: init 2024-03-29T11:44:48.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:48.363 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 10 monclient(hunting): picked mon.noname-b con 0x7f44581457f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 10 monclient(hunting): picked mon.noname-a con 0x7f445813f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 10 monclient(hunting): picked mon.noname-c con 0x7f44580a9d70 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445659c640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f44581457f0 0x7f445813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:38876/0 (socket says 172.21.15.145:38876) 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445659c640 1 -- 172.21.15.145:0/892432835 learned_addr learned my addr 172.21.15.145:0/892432835 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 1 -- 172.21.15.145:0/892432835 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580aaef0 con 0x7f445813f650 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 1 -- 172.21.15.145:0/892432835 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4458145bd0 con 0x7f44580a9d70 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 1 -- 172.21.15.145:0/892432835 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4458145eb0 con 0x7f44581457f0 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:48.364 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.360+0000 7f445cc40640 10 monclient(hunting): authenticate will time out at 1477.667724s 2024-03-29T11:44:48.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146854209 0 0) 0x7f4458145eb0 con 0x7f44581457f0 2024-03-29T11:44:48.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:48.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:48.366 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4440003520 con 0x7f44581457f0 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890641578 0 0) 0x7f44580aaef0 con 0x7f445813f650 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 10 monclient(hunting): _negotiate 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4458145eb0 con 0x7f445813f650 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580744445 0 0) 0x7f4458145eb0 con 0x7f445813f650 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 >> v1:172.21.15.145:6790/0 conn(0x7f44581457f0 legacy=0x7f445813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 >> v1:172.21.15.203:6789/0 conn(0x7f44580a9d70 legacy=0x7f445814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.361+0000 7f445559a640 10 monclient: _finish_hunting 0 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 1 monclient: found mon.noname-a 2024-03-29T11:44:48.367 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4458146340 con 0x7f445813f650 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 10 monclient: _finish_auth 0 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 10 monclient: _check_auth_tickets 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4448001670 con 0x7f445813f650 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 10 monclient: dump: 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:48.368 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445cc40640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445cc40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4448001890 con 0x7f445813f650 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.362+0000 7f445559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.363+0000 7f445559a640 1 -- 172.21.15.145:0/892432835 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4448001ba0 con 0x7f445813f650 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.363+0000 7f445cc40640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.363+0000 7f445cc40640 4 set_mon_vals no callback set 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.363+0000 7f445559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.363+0000 7f445559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.363+0000 7f445559a640 10 monclient: dump: 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:48.369 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:48.370 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.370 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.370 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:48.370 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:48.370 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:48.370 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:48.370 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:48.370 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:48.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.363+0000 7f445cc40640 10 monclient: shutdown 2024-03-29T11:44:48.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.363+0000 7f445cc40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:48.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.363+0000 7f445cc40640 1 -- 172.21.15.145:0/892432835 >> v1:172.21.15.145:6789/0 conn(0x7f445813f650 legacy=0x7f44580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:48.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.364+0000 7f445cc40640 1 -- 172.21.15.145:0/892432835 shutdown_connections 2024-03-29T11:44:48.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.364+0000 7f445cc40640 1 -- 172.21.15.145:0/892432835 >> 172.21.15.145:0/892432835 conn(0x7f4458066a00 msgr2=0x7f4458066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:48.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.364+0000 7f445cc40640 1 -- 172.21.15.145:0/892432835 shutdown_connections 2024-03-29T11:44:48.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.364+0000 7f445cc40640 1 -- 172.21.15.145:0/892432835 wait complete. 2024-03-29T11:44:48.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.365+0000 7f445cc40640 10 monclient: build_initial_monmap 2024-03-29T11:44:48.377 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.365+0000 7f445cc40640 10 monclient: monmap: 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.365+0000 7f445cc40640 1 librados: starting msgr at 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.365+0000 7f445cc40640 1 librados: starting objecter 2024-03-29T11:44:48.378 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.366+0000 7f445cc40640 1 Processor -- start 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.366+0000 7f445cc40640 1 -- start start 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.366+0000 7f445cc40640 1 librados: setting wanted keys 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.366+0000 7f445cc40640 1 librados: calling monclient init 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.366+0000 7f445cc40640 10 monclient: init 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 10 monclient(hunting): picked mon.noname-c con 0x7f44580a9d70 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 10 monclient(hunting): picked mon.noname-a con 0x7f445813f650 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 10 monclient(hunting): picked mon.noname-b con 0x7f44581457f0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580adb60 con 0x7f445813f650 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f445813a7e0 con 0x7f44581457f0 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44580ae700 con 0x7f44580a9d70 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445659c640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f44580a9d70 0x7f44580adff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:39492/0 (socket says 172.21.15.145:39492) 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445659c640 1 -- 172.21.15.145:0/679445106 learned_addr learned my addr 172.21.15.145:0/679445106 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:48.379 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f445cc40640 10 monclient(hunting): authenticate will time out at 1477.674659s 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 client.?.objecter ms_handle_connect 0x7f445813f650 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 client.?.objecter ms_handle_connect 0x7f44581457f0 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 client.?.objecter ms_handle_connect 0x7f44580a9d70 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660267402 0 0) 0x7f445813a7e0 con 0x7f44581457f0 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 10 monclient(hunting): my global_id is 4573 2024-03-29T11:44:48.380 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.367+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4438004eb0 con 0x7f44581457f0 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479891178 0 0) 0x7f44580adb60 con 0x7f445813f650 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f445813a7e0 con 0x7f445813f650 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4276194185 0 0) 0x7f4438004eb0 con 0x7f44581457f0 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f44580adb60 con 0x7f44581457f0 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4450001b40 con 0x7f44581457f0 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:48.381 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2037837195 0 0) 0x7f445813a7e0 con 0x7f445813f650 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4438004eb0 con 0x7f445813f650 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4448001530 con 0x7f445813f650 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): dump: 2024-03-29T11:44:48.382 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392204167 0 0) 0x7f44580ae700 con 0x7f44580a9d70 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:48.383 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f445813a7e0 con 0x7f44580a9d70 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1491705716 0 0) 0x7f44580adb60 con 0x7f44581457f0 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.368+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 >> v1:172.21.15.145:6790/0 conn(0x7f44580a9d70 legacy=0x7f44580adff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 >> v1:172.21.15.145:6789/0 conn(0x7f445813f650 legacy=0x7f44581399a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f4446ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f4446ffd640 1 monclient: found mon.b 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f4446ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f4446ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44580ae890 con 0x7f44581457f0 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f4446ffd640 10 monclient: _finish_auth 0 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f4446ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f4446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f445cc40640 5 monclient: authenticate success, global_id 4573 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f445cc40640 10 monclient: _renew_subs 2024-03-29T11:44:48.384 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f445cc40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44580afed0 con 0x7f44581457f0 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f445cc40640 10 client.4573.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f445cc40640 10 monclient: _renew_subs 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f445cc40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f44580b16e0 con 0x7f44581457f0 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.369+0000 7f445cc40640 1 librados: init done 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.370+0000 7f445cc40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.370+0000 7f445cc40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.370+0000 7f445cc40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.370+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44580b7460 con 0x7f44581457f0 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.373+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4450002050 con 0x7f44581457f0 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.373+0000 7f4446ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.373+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4450003b10 con 0x7f44581457f0 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.373+0000 7f4446ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:48.385 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.373+0000 7f4446ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.373+0000 7f4446ffd640 10 monclient: dump: 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:48.386 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.373+0000 7f4447fff640 4 set_mon_vals no callback set 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.374+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f4450019b60 con 0x7f44581457f0 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.375+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f4450083ad0 con 0x7f44581457f0 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.375+0000 7f4446ffd640 10 client.4573.objecter ms_dispatch 0x7f4458000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.375+0000 7f4446ffd640 3 client.4573.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.375+0000 7f4446ffd640 3 client.4573.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.375+0000 7f4446ffd640 20 client.4573.objecter dump_active .. 0 homeless 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.375+0000 7f445659c640 10 monclient: get_auth_request con 0x7f443806df50 auth_method 0 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.375+0000 7f4446ffd640 10 client.4573.objecter ms_handle_connect 0x7f443806df50 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.380+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f445004f290 con 0x7f44581457f0 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.380+0000 7f4446ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:48.388 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.380+0000 7f4446ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:48.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.728+0000 7f445cc40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:48.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.728+0000 7f445cc40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:48.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.728+0000 7f445cc40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:44:48.732 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.728+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44580b7180 con 0x7f44581457f0 2024-03-29T11:44:48.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.732+0000 7f4446ffd640 1 -- 172.21.15.145:0/679445106 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f4450051390 con 0x7f44581457f0 2024-03-29T11:44:48.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.732+0000 7f4446ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:48.735 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.732+0000 7f4446ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:48.735 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:44:48.736 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:48.738 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.734+0000 7f445cc40640 10 librados: watch_flush enter 2024-03-29T11:44:48.738 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.734+0000 7f445cc40640 10 librados: watch_flush exit 2024-03-29T11:44:48.738 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.734+0000 7f445cc40640 20 client.4573.objecter shutdown clearing up homeless session... 2024-03-29T11:44:48.738 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.734+0000 7f445cc40640 10 client.4573.objecter successfully canceled tick 2024-03-29T11:44:48.738 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.734+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 >> v1:172.21.15.145:6816/227372239 conn(0x7f443806df50 legacy=0x7f4438070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:48.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.735+0000 7f445cc40640 10 monclient: shutdown 2024-03-29T11:44:48.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.735+0000 7f445cc40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:48.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.735+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 >> v1:172.21.15.203:6789/0 conn(0x7f44581457f0 legacy=0x7f445813a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:48.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.735+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 shutdown_connections 2024-03-29T11:44:48.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.735+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 >> 172.21.15.145:0/679445106 conn(0x7f4458066a00 msgr2=0x7f445813ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:48.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.735+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 shutdown_connections 2024-03-29T11:44:48.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.735+0000 7f445cc40640 1 -- 172.21.15.145:0/679445106 wait complete. 2024-03-29T11:44:48.739 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:48.735+0000 7f445cc40640 1 librados: shutdown 2024-03-29T11:44:48.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:44:55.559 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.701+0000 7f7922918640 10 monclient: get_monmap_and_config 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.701+0000 7f7922918640 10 monclient: build_initial_monmap 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.701+0000 7f7922918640 10 monclient: monmap: 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:44:55.702184+0000 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:44:55.702184+0000 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:44:55.706 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:44:55.707 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:55.707 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.703+0000 7f7922918640 1 Processor -- start 2024-03-29T11:44:55.707 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.703+0000 7f7922918640 1 -- start start 2024-03-29T11:44:55.707 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.703+0000 7f7922918640 10 monclient: init 2024-03-29T11:44:55.707 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7922918640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:55.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7922918640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:44:55.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7922918640 10 monclient(hunting): picked mon.noname-c con 0x7f791c0b4920 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:55.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7922918640 10 monclient(hunting): picked mon.noname-a con 0x7f791c0b4200 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:55.710 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7922918640 10 monclient(hunting): picked mon.noname-b con 0x7f791c13f9a0 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7922918640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c0b0930 con 0x7f791c0b4200 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7922918640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c0af460 con 0x7f791c0b4920 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7921115640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f791c0b4200 0x7f791c0b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:51468/0 (socket says 172.21.15.145:51468) 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7921115640 1 -- 172.21.15.145:0/2320753336 learned_addr learned my addr 172.21.15.145:0/2320753336 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.704+0000 7f7922117640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f791c13f9a0 0x7f791c14d8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:48506/0 (socket says 172.21.15.145:48506) 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7922918640 1 -- 172.21.15.145:0/2320753336 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c0aeda0 con 0x7f791c13f9a0 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7921115640 1 -- 172.21.15.145:0/2320753336 >> v1:172.21.15.145:6789/0 conn(0x7f791c0b4200 legacy=0x7f791c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7921115640 1 -- 172.21.15.145:0/2320753336 >> v1:172.21.15.145:6789/0 conn(0x7f791c0b4200 legacy=0x7f791c0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7921115640 1 --1- 172.21.15.145:0/2320753336 >> v1:172.21.15.145:6789/0 conn(0x7f791c0b4200 0x7f791c0b0fc0 :-1 s=OPENED pgs=955283 cs=1 l=1).handle_message read tag failed 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7921115640 1 --1- 172.21.15.145:0/2320753336 >> v1:172.21.15.145:6789/0 conn(0x7f791c0b4200 0x7f791c0b0fc0 :-1 s=OPENED pgs=955283 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7922918640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7922918640 10 monclient(hunting): authenticate will time out at 1485.012308s 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7920914640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.145:6789/0 2024-03-29T11:44:55.711 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7920914640 1 -- 172.21.15.145:0/2320753336 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160255931 0 0) 0x7f791c0aeda0 con 0x7f791c13f9a0 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7920914640 10 monclient(hunting): _negotiate 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7920914640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7920914640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7920914640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7920914640 1 -- 172.21.15.145:0/2320753336 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7908003520 con 0x7f791c13f9a0 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7920914640 1 -- 172.21.15.145:0/2320753336 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4176858450 0 0) 0x7f7908003520 con 0x7f791c13f9a0 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.705+0000 7f7920914640 1 -- 172.21.15.145:0/2320753336 >> v1:172.21.15.203:6789/0 conn(0x7f791c0b4920 legacy=0x7f791c13f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 1 -- 172.21.15.145:0/2320753336 >> v1:172.21.15.145:6789/0 conn(0x7f791c0b4200 legacy=0x7f791c0b0fc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 10 monclient: _finish_hunting 0 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 1 monclient: found mon.noname-b 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 1 -- 172.21.15.145:0/2320753336 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c0af980 con 0x7f791c13f9a0 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 10 monclient: _finish_auth 0 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 10 monclient: _check_auth_tickets 2024-03-29T11:44:55.712 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 1 -- 172.21.15.145:0/2320753336 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f7914001a10 con 0x7f791c13f9a0 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 10 monclient: dump: 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:55.713 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7922918640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7922918640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 1 -- 172.21.15.145:0/2320753336 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7914001b70 con 0x7f791c13f9a0 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 1 -- 172.21.15.145:0/2320753336 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f7914001fc0 con 0x7f791c13f9a0 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.706+0000 7f7920914640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.707+0000 7f7920914640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.707+0000 7f7920914640 10 monclient: dump: 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:55.714 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:55.715 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:55.715 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:55.715 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:55.715 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:55.715 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:55.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.707+0000 7f7922918640 10 monclient: get_monmap_and_config success 2024-03-29T11:44:55.715 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.707+0000 7f7922918640 4 set_mon_vals no callback set 2024-03-29T11:44:55.720 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.707+0000 7f7922918640 10 monclient: shutdown 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.707+0000 7f7922918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.707+0000 7f7922918640 1 -- 172.21.15.145:0/2320753336 >> v1:172.21.15.145:6790/0 conn(0x7f791c13f9a0 legacy=0x7f791c14d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.708+0000 7f7922918640 1 -- 172.21.15.145:0/2320753336 shutdown_connections 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.708+0000 7f7922918640 1 -- 172.21.15.145:0/2320753336 >> 172.21.15.145:0/2320753336 conn(0x7f791c06d230 msgr2=0x7f791c066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.708+0000 7f7922918640 1 -- 172.21.15.145:0/2320753336 shutdown_connections 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.708+0000 7f7922918640 1 -- 172.21.15.145:0/2320753336 wait complete. 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.709+0000 7f7922918640 10 monclient: build_initial_monmap 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.709+0000 7f7922918640 10 monclient: monmap: 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:44:55.721 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.709+0000 7f7922918640 1 librados: starting msgr at 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.709+0000 7f7922918640 1 librados: starting objecter 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.710+0000 7f7922918640 1 Processor -- start 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.710+0000 7f7922918640 1 -- start start 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.710+0000 7f7922918640 1 librados: setting wanted keys 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.710+0000 7f7922918640 1 librados: calling monclient init 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.710+0000 7f7922918640 10 monclient: init 2024-03-29T11:44:55.722 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.710+0000 7f7922918640 10 monclient: _reopen_session rank -1 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.710+0000 7f7922918640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.710+0000 7f7922918640 10 monclient(hunting): picked mon.noname-a con 0x7f791c0b4200 addr v1:172.21.15.145:6789/0 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f7922918640 10 monclient(hunting): picked mon.noname-c con 0x7f791c0b4920 addr v1:172.21.15.145:6790/0 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f7922918640 10 monclient(hunting): picked mon.noname-b con 0x7f791c13f9a0 addr v1:172.21.15.203:6789/0 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f7922918640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c0aac20 con 0x7f791c0b4200 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f7922918640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c0aae90 con 0x7f791c13f9a0 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f7922918640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c1d6ec0 con 0x7f791c0b4920 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f7922918640 10 monclient(hunting): _renew_subs 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f7922918640 10 monclient(hunting): authenticate will time out at 1485.018246s 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f7921115640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f791c0b4920 0x7f791c1d73e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:48974/0 (socket says 172.21.15.145:48974) 2024-03-29T11:44:55.723 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f7921115640 1 -- 172.21.15.145:0/1395534247 learned_addr learned my addr 172.21.15.145:0/1395534247 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f79127fc640 10 client.?.objecter ms_handle_connect 0x7f791c0b4920 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f79127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f79127fc640 10 client.?.objecter ms_handle_connect 0x7f791c13f9a0 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f79127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f79127fc640 10 client.?.objecter ms_handle_connect 0x7f791c0b4200 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f79127fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.711+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281226596 0 0) 0x7f791c1d6ec0 con 0x7f791c0b4920 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7900004eb0 con 0x7f791c0b4920 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215704630 0 0) 0x7f791c0aae90 con 0x7f791c13f9a0 2024-03-29T11:44:55.724 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): my global_id is 4576 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f791c1d6ec0 con 0x7f791c13f9a0 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3644285977 0 0) 0x7f7900004eb0 con 0x7f791c0b4920 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f791c0aae90 con 0x7f791c0b4920 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f790c0018e0 con 0x7f791c0b4920 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): dump: 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:55.725 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350878337 0 0) 0x7f791c0aac20 con 0x7f791c0b4200 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:44:55.726 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79000057f0 con 0x7f791c0b4200 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1428466550 0 0) 0x7f791c0aae90 con 0x7f791c0b4920 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 >> v1:172.21.15.203:6789/0 conn(0x7f791c13f9a0 legacy=0x7f791c138de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 >> v1:172.21.15.145:6789/0 conn(0x7f791c0b4200 legacy=0x7f791c1458b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 monclient: found mon.c 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.712+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c1d7af0 con 0x7f791c0b4920 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f79127fc640 10 monclient: _finish_auth 0 2024-03-29T11:44:55.727 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f79127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f79127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7922918640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7922918640 10 monclient: _renew_subs 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7922918640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791c143320 con 0x7f791c0b4920 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7922918640 10 client.4610.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7922918640 10 monclient: _renew_subs 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7922918640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f791c1396d0 con 0x7f791c0b4920 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7922918640 1 librados: init done 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f790c0047e0 con 0x7f791c0b4920 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f79127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.713+0000 7f7913fff640 4 set_mon_vals no callback set 2024-03-29T11:44:55.728 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.714+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f790c0018e0 con 0x7f791c0b4920 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.714+0000 7f79127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.714+0000 7f79127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.714+0000 7f79127fc640 10 monclient: dump: 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.714+0000 7f7922918640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.714+0000 7f7922918640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:55.729 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.714+0000 7f7922918640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:55.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.714+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7904005080 con 0x7f791c0b4920 2024-03-29T11:44:55.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.715+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f790c004ad0 con 0x7f791c0b4920 2024-03-29T11:44:55.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.716+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7f790c084480 con 0x7f791c0b4920 2024-03-29T11:44:55.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.716+0000 7f79127fc640 10 client.4610.objecter ms_dispatch 0x7f791c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:44:55.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.716+0000 7f79127fc640 3 client.4610.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:44:55.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.716+0000 7f79127fc640 3 client.4610.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:44:55.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.716+0000 7f79127fc640 20 client.4610.objecter dump_active .. 0 homeless 2024-03-29T11:44:55.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.716+0000 7f7921916640 10 monclient: get_auth_request con 0x7f790006e330 auth_method 0 2024-03-29T11:44:55.730 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.716+0000 7f79127fc640 10 client.4610.objecter ms_handle_connect 0x7f790006e330 2024-03-29T11:44:55.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.726+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f790c04fcd0 con 0x7f791c0b4920 2024-03-29T11:44:55.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.726+0000 7f79127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:44:55.731 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:55.726+0000 7f79127fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:44:56.090 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.086+0000 7f7922918640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:56.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.086+0000 7f7922918640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:56.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.087+0000 7f7922918640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:44:56.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.087+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7904005330 con 0x7f791c0b4920 2024-03-29T11:44:56.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.087+0000 7f79127fc640 1 -- 172.21.15.145:0/1395534247 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7f790c04fcd0 con 0x7f791c0b4920 2024-03-29T11:44:56.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.087+0000 7f79127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:44:56.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.087+0000 7f79127fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:44:56.092 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:44:56.092 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:44:56.094 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.091+0000 7f7922918640 10 librados: watch_flush enter 2024-03-29T11:44:56.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.091+0000 7f7922918640 10 librados: watch_flush exit 2024-03-29T11:44:56.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.091+0000 7f7922918640 20 client.4610.objecter shutdown clearing up homeless session... 2024-03-29T11:44:56.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.091+0000 7f7922918640 10 client.4610.objecter successfully canceled tick 2024-03-29T11:44:56.096 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.091+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 >> v1:172.21.15.145:6816/227372239 conn(0x7f790006e330 legacy=0x7f79000707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:56.096 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.091+0000 7f7922918640 10 monclient: shutdown 2024-03-29T11:44:56.096 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.091+0000 7f7922918640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:44:56.096 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.091+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 >> v1:172.21.15.145:6790/0 conn(0x7f791c0b4920 legacy=0x7f791c1d73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:44:56.096 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.092+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 shutdown_connections 2024-03-29T11:44:56.096 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.092+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 >> 172.21.15.145:0/1395534247 conn(0x7f791c06d230 msgr2=0x7f791c13c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:44:56.096 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.092+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 shutdown_connections 2024-03-29T11:44:56.096 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.092+0000 7f7922918640 1 -- 172.21.15.145:0/1395534247 wait complete. 2024-03-29T11:44:56.096 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:44:56.092+0000 7f7922918640 1 librados: shutdown 2024-03-29T11:44:56.110 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:02.917 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.065+0000 7fe9f759e640 10 monclient: get_monmap_and_config 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.065+0000 7fe9f759e640 10 monclient: build_initial_monmap 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.065+0000 7fe9f759e640 10 monclient: monmap: 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:45:03.067110+0000 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:45:03.067110+0000 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:03.071 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:45:03.072 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:45:03.072 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:03.073 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.069+0000 7fe9f759e640 1 Processor -- start 2024-03-29T11:45:03.074 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.069+0000 7fe9f759e640 1 -- start start 2024-03-29T11:45:03.074 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.069+0000 7fe9f759e640 10 monclient: init 2024-03-29T11:45:03.074 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:03.075 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:03.075 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f80a9750 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:03.075 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f80ae220 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:03.075 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 10 monclient(hunting): picked mon.noname-c con 0x7fe9f813b9d0 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:03.075 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f8142490 con 0x7fe9f80ae220 2024-03-29T11:45:03.075 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f80ac130 con 0x7fe9f813b9d0 2024-03-29T11:45:03.075 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f80ac410 con 0x7fe9f80a9750 2024-03-29T11:45:03.075 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:03.075 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.070+0000 7fe9f759e640 10 monclient(hunting): authenticate will time out at 1492.378527s 2024-03-29T11:45:03.079 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.075+0000 7fe9f5d9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fe9f80ae220 0x7fe9f813b2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:34935/0 (socket says 172.21.15.145:34935) 2024-03-29T11:45:03.079 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.075+0000 7fe9f5d9b640 1 -- 172.21.15.145:0/3569817076 learned_addr learned my addr 172.21.15.145:0/3569817076 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:03.082 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.078+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1685884131 0 0) 0x7fe9f8142490 con 0x7fe9f80ae220 2024-03-29T11:45:03.083 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.078+0000 7fe9f559a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:03.083 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.078+0000 7fe9f559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:03.083 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.078+0000 7fe9f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:03.083 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.078+0000 7fe9f559a640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:45:03.083 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.078+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9e0003520 con 0x7fe9f80ae220 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.079+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484165448 0 0) 0x7fe9f80ac410 con 0x7fe9f80a9750 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.079+0000 7fe9f559a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.079+0000 7fe9f559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.079+0000 7fe9f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9f8142490 con 0x7fe9f80a9750 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1002591807 0 0) 0x7fe9e0003520 con 0x7fe9f80ae220 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 >> v1:172.21.15.145:6790/0 conn(0x7fe9f80a9750 legacy=0x7fe9f8141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 >> v1:172.21.15.203:6789/0 conn(0x7fe9f813b9d0 legacy=0x7fe9f8149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:03.088 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 10 monclient: _finish_hunting 0 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 1 monclient: found mon.noname-a 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f80afa40 con 0x7fe9f80ae220 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 10 monclient: _finish_auth 0 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 10 monclient: _check_auth_tickets 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe9e8001690 con 0x7fe9f80ae220 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f759e640 5 monclient: authenticate success, global_id 4584 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.080+0000 7fe9f759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:03.089 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f559a640 10 monclient: dump: 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:03.090 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:03.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f759e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:03.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9e80018b0 con 0x7fe9f80ae220 2024-03-29T11:45:03.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:03.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f559a640 1 -- 172.21.15.145:0/3569817076 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe9e8001bc0 con 0x7fe9f80ae220 2024-03-29T11:45:03.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:03.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f759e640 4 set_mon_vals no callback set 2024-03-29T11:45:03.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:03.091 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.081+0000 7fe9f559a640 10 monclient: dump: 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.082+0000 7fe9f759e640 10 monclient: shutdown 2024-03-29T11:45:03.092 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.082+0000 7fe9f759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:03.093 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.082+0000 7fe9f759e640 1 -- 172.21.15.145:0/3569817076 >> v1:172.21.15.145:6789/0 conn(0x7fe9f80ae220 legacy=0x7fe9f813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:03.093 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.083+0000 7fe9f759e640 1 -- 172.21.15.145:0/3569817076 shutdown_connections 2024-03-29T11:45:03.093 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.083+0000 7fe9f759e640 1 -- 172.21.15.145:0/3569817076 >> 172.21.15.145:0/3569817076 conn(0x7fe9f80673e0 msgr2=0x7fe9f80b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:03.093 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.083+0000 7fe9f759e640 1 -- 172.21.15.145:0/3569817076 shutdown_connections 2024-03-29T11:45:03.093 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.083+0000 7fe9f759e640 1 -- 172.21.15.145:0/3569817076 wait complete. 2024-03-29T11:45:03.093 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.084+0000 7fe9f759e640 10 monclient: build_initial_monmap 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.084+0000 7fe9f759e640 10 monclient: monmap: 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:45:03.094 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:03.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.084+0000 7fe9f759e640 1 librados: starting msgr at 2024-03-29T11:45:03.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.084+0000 7fe9f759e640 1 librados: starting objecter 2024-03-29T11:45:03.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.085+0000 7fe9f759e640 1 Processor -- start 2024-03-29T11:45:03.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.085+0000 7fe9f759e640 1 -- start start 2024-03-29T11:45:03.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.085+0000 7fe9f759e640 1 librados: setting wanted keys 2024-03-29T11:45:03.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.085+0000 7fe9f759e640 1 librados: calling monclient init 2024-03-29T11:45:03.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.085+0000 7fe9f759e640 10 monclient: init 2024-03-29T11:45:03.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.088+0000 7fe9f759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:03.095 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.088+0000 7fe9f759e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f759e640 10 monclient(hunting): picked mon.noname-b con 0x7fe9f80a9750 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f759e640 10 monclient(hunting): picked mon.noname-a con 0x7fe9f80ae220 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f659c640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7fe9f80a9750 0x7fe9f81e3d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:44001/0 (socket says 172.21.15.145:44001) 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f659c640 1 -- 172.21.15.145:0/1715036839 learned_addr learned my addr 172.21.15.145:0/1715036839 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f759e640 10 monclient(hunting): picked mon.noname-c con 0x7fe9f813b9d0 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f81e4930 con 0x7fe9f80ae220 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9e6ffd640 10 client.?.objecter ms_handle_connect 0x7fe9f80a9750 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f81e4460 con 0x7fe9f80a9750 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f81e45f0 con 0x7fe9f813b9d0 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.093+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099448049 0 0) 0x7fe9f81e4460 con 0x7fe9f80a9750 2024-03-29T11:45:03.098 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.092+0000 7fe9f759e640 10 monclient(hunting): authenticate will time out at 1492.400990s 2024-03-29T11:45:03.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.093+0000 7fe9e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:03.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.093+0000 7fe9e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:03.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.093+0000 7fe9e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:03.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.093+0000 7fe9e6ffd640 10 monclient(hunting): my global_id is 4582 2024-03-29T11:45:03.099 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.093+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9d8004eb0 con 0x7fe9f80a9750 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.094+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3271535478 0 0) 0x7fe9d8004eb0 con 0x7fe9f80a9750 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.094+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9f81e4460 con 0x7fe9f80a9750 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.094+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe9ec001a10 con 0x7fe9f80a9750 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.094+0000 7fe9e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.094+0000 7fe9e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.094+0000 7fe9e6ffd640 10 monclient(hunting): dump: 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:03.102 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:03.103 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:03.103 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:03.103 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:03.105 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:03.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.094+0000 7fe9e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:03.105 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.095+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3181306546 0 0) 0x7fe9f81e4460 con 0x7fe9f80a9750 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.095+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 >> v1:172.21.15.145:6790/0 conn(0x7fe9f813b9d0 legacy=0x7fe9f81e7250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.095+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 >> v1:172.21.15.145:6789/0 conn(0x7fe9f80ae220 legacy=0x7fe9f81409a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9e6ffd640 1 monclient: found mon.b 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f81370b0 con 0x7fe9f80a9750 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9e6ffd640 10 client.?.objecter ms_handle_connect 0x7fe9f813b9d0 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9f759e640 5 monclient: authenticate success, global_id 4582 2024-03-29T11:45:03.106 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9f759e640 10 monclient: _renew_subs 2024-03-29T11:45:03.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9f759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:03.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f8137360 con 0x7fe9f80a9750 2024-03-29T11:45:03.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.096+0000 7fe9f759e640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:03.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.097+0000 7fe9f759e640 10 monclient: _renew_subs 2024-03-29T11:45:03.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.097+0000 7fe9f759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:03.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.097+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f8141290 con 0x7fe9f80a9750 2024-03-29T11:45:03.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.097+0000 7fe9f759e640 1 librados: init done 2024-03-29T11:45:03.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.097+0000 7fe9e6ffd640 10 client.4582.objecter resend_mon_ops 2024-03-29T11:45:03.107 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.098+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9ec001b70 con 0x7fe9f80a9750 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.098+0000 7fe9f759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.098+0000 7fe9f759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.098+0000 7fe9f759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.098+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f804f730 con 0x7fe9f80a9750 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.099+0000 7fe9e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.099+0000 7fe9f4d99640 4 set_mon_vals no callback set 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.101+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7fe9ec003a60 con 0x7fe9f80a9750 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.101+0000 7fe9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.101+0000 7fe9e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:03.108 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.101+0000 7fe9e6ffd640 10 monclient: dump: 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:03.109 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.101+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7fe9ec003c80 con 0x7fe9f80a9750 2024-03-29T11:45:03.111 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.104+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2657814457 0 0) 0x7fe9ec084ba0 con 0x7fe9f80a9750 2024-03-29T11:45:03.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.104+0000 7fe9e6ffd640 10 client.4582.objecter ms_dispatch 0x7fe9f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:45:03.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.104+0000 7fe9e6ffd640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:45:03.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.104+0000 7fe9e6ffd640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:45:03.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.104+0000 7fe9e6ffd640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-29T11:45:03.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.105+0000 7fe9f5d9b640 10 monclient: get_auth_request con 0x7fe9d806dc70 auth_method 0 2024-03-29T11:45:03.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.106+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9ec050500 con 0x7fe9f80a9750 2024-03-29T11:45:03.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.106+0000 7fe9e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:03.112 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.106+0000 7fe9e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:03.113 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.108+0000 7fe9e6ffd640 10 client.4582.objecter ms_handle_connect 0x7fe9d806dc70 2024-03-29T11:45:03.433 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.427+0000 7fe9f759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:03.433 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.427+0000 7fe9f759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:03.433 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.427+0000 7fe9f759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:03.434 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.427+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9f80abb70 con 0x7fe9f80a9750 2024-03-29T11:45:03.436 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:45:03.436 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":10,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:35:07.201884+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:35:07.201884+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:03.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.430+0000 7fe9e6ffd640 1 -- 172.21.15.145:0/1715036839 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2327222713) 0x7fe9ec052430 con 0x7fe9f80a9750 2024-03-29T11:45:03.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.430+0000 7fe9e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:03.437 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.430+0000 7fe9e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:03.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.436+0000 7fe9f759e640 10 librados: watch_flush enter 2024-03-29T11:45:03.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.436+0000 7fe9f759e640 10 librados: watch_flush exit 2024-03-29T11:45:03.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.436+0000 7fe9f759e640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-29T11:45:03.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.436+0000 7fe9f759e640 10 client.4582.objecter successfully canceled tick 2024-03-29T11:45:03.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.436+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 >> v1:172.21.15.145:6816/227372239 conn(0x7fe9d806dc70 legacy=0x7fe9d8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:03.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.436+0000 7fe9f759e640 10 monclient: shutdown 2024-03-29T11:45:03.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.436+0000 7fe9f759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:03.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.437+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 >> v1:172.21.15.203:6789/0 conn(0x7fe9f80a9750 legacy=0x7fe9f81e3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:03.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.439+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 shutdown_connections 2024-03-29T11:45:03.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.439+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 >> 172.21.15.145:0/1715036839 conn(0x7fe9f80673e0 msgr2=0x7fe9f813b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:03.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.440+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 shutdown_connections 2024-03-29T11:45:03.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.444+0000 7fe9f759e640 1 -- 172.21.15.145:0/1715036839 wait complete. 2024-03-29T11:45:03.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:03.444+0000 7fe9f759e640 1 librados: shutdown 2024-03-29T11:45:03.473 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:10.280 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:10.442 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.435+0000 7f49207ac640 10 monclient: get_monmap_and_config 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.435+0000 7f49207ac640 10 monclient: build_initial_monmap 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.435+0000 7f49207ac640 10 monclient: monmap: 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:45:10.437194+0000 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:45:10.437194+0000 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.436+0000 7f49207ac640 1 Processor -- start 2024-03-29T11:45:10.443 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.437+0000 7f49207ac640 1 -- start start 2024-03-29T11:45:10.444 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.437+0000 7f49207ac640 10 monclient: init 2024-03-29T11:45:10.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:10.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:10.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 10 monclient(hunting): picked mon.noname-b con 0x7f4910029ed0 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:10.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 10 monclient(hunting): picked mon.noname-a con 0x7f491002b0c0 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:10.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 10 monclient(hunting): picked mon.noname-c con 0x7f49100c9590 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:10.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491002a2b0 con 0x7f491002b0c0 2024-03-29T11:45:10.447 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49100331e0 con 0x7f49100c9590 2024-03-29T11:45:10.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491002c120 con 0x7f4910029ed0 2024-03-29T11:45:10.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:10.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.438+0000 7f49207ac640 10 monclient(hunting): authenticate will time out at 1499.746754s 2024-03-29T11:45:10.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.442+0000 7f491ed49640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f49100c9590 0x7f49100d74e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:33015/0 (socket says 172.21.15.145:33015) 2024-03-29T11:45:10.448 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.442+0000 7f491ed49640 1 -- 172.21.15.145:0/1684650180 learned_addr learned my addr 172.21.15.145:0/1684650180 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:10.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.444+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853298876 0 0) 0x7f491002c120 con 0x7f4910029ed0 2024-03-29T11:45:10.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.444+0000 7f491d546640 10 monclient(hunting): _negotiate 2024-03-29T11:45:10.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.444+0000 7f491d546640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:10.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.444+0000 7f491d546640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:10.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.444+0000 7f491d546640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:45:10.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.444+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904003520 con 0x7f4910029ed0 2024-03-29T11:45:10.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.445+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350784932 0 0) 0x7f49100331e0 con 0x7f49100c9590 2024-03-29T11:45:10.450 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.445+0000 7f491d546640 10 monclient(hunting): _negotiate 2024-03-29T11:45:10.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.445+0000 7f491d546640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:10.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.445+0000 7f491d546640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:10.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.445+0000 7f491d546640 10 monclient(hunting): my global_id is 4585 2024-03-29T11:45:10.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.445+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f491002c120 con 0x7f49100c9590 2024-03-29T11:45:10.451 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446998490 0 0) 0x7f491002a2b0 con 0x7f491002b0c0 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient(hunting): _negotiate 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49100331e0 con 0x7f491002b0c0 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3102853449 0 0) 0x7f491002c120 con 0x7f49100c9590 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 >> v1:172.21.15.145:6790/0 conn(0x7f4910029ed0 legacy=0x7f49100cf140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 >> v1:172.21.15.145:6789/0 conn(0x7f491002b0c0 legacy=0x7f49100c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: _finish_hunting 0 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 monclient: found mon.noname-c 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491002a460 con 0x7f49100c9590 2024-03-29T11:45:10.452 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: _finish_auth 0 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: _check_auth_tickets 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f491804f730 con 0x7f49100c9590 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: dump: 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:10.453 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49180008d0 con 0x7f49100c9590 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 1 -- 172.21.15.145:0/1684650180 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f49180758b0 con 0x7f49100c9590 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.446+0000 7f491d546640 10 monclient: dump: 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:10.454 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:10.455 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:10.455 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.448+0000 7f49207ac640 5 monclient: authenticate success, global_id 4585 2024-03-29T11:45:10.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.448+0000 7f49207ac640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:10.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.448+0000 7f49207ac640 4 set_mon_vals no callback set 2024-03-29T11:45:10.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.449+0000 7f49207ac640 10 monclient: shutdown 2024-03-29T11:45:10.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.449+0000 7f49207ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:10.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.449+0000 7f49207ac640 1 -- 172.21.15.145:0/1684650180 >> v1:172.21.15.203:6789/0 conn(0x7f49100c9590 legacy=0x7f49100d74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:10.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.449+0000 7f49207ac640 1 -- 172.21.15.145:0/1684650180 shutdown_connections 2024-03-29T11:45:10.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.450+0000 7f49207ac640 1 -- 172.21.15.145:0/1684650180 >> 172.21.15.145:0/1684650180 conn(0x7f4910005910 msgr2=0x7f4910005d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:10.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.450+0000 7f49207ac640 1 -- 172.21.15.145:0/1684650180 shutdown_connections 2024-03-29T11:45:10.456 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.450+0000 7f49207ac640 1 -- 172.21.15.145:0/1684650180 wait complete. 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.451+0000 7f49207ac640 10 monclient: build_initial_monmap 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.451+0000 7f49207ac640 10 monclient: monmap: 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:45:10.458 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.451+0000 7f49207ac640 1 librados: starting msgr at 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.451+0000 7f49207ac640 1 librados: starting objecter 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.451+0000 7f49207ac640 1 Processor -- start 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.451+0000 7f49207ac640 1 -- start start 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.451+0000 7f49207ac640 1 librados: setting wanted keys 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.451+0000 7f49207ac640 1 librados: calling monclient init 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.451+0000 7f49207ac640 10 monclient: init 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 10 monclient(hunting): picked mon.noname-a con 0x7f4910029ed0 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 10 monclient(hunting): picked mon.noname-c con 0x7f491002b0c0 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 10 monclient(hunting): picked mon.noname-b con 0x7f49100c9590 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4910160a80 con 0x7f4910029ed0 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491002e5c0 con 0x7f49100c9590 2024-03-29T11:45:10.459 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4910161900 con 0x7f491002b0c0 2024-03-29T11:45:10.460 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:10.460 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.452+0000 7f49207ac640 10 monclient(hunting): authenticate will time out at 1499.760571s 2024-03-29T11:45:10.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.456+0000 7f491dd47640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f491002b0c0 0x7f49100c1790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:33977/0 (socket says 172.21.15.145:33977) 2024-03-29T11:45:10.461 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.456+0000 7f491dd47640 1 -- 172.21.15.145:0/1731326872 learned_addr learned my addr 172.21.15.145:0/1731326872 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:10.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 10 client.?.objecter ms_handle_connect 0x7f491002b0c0 2024-03-29T11:45:10.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:10.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 10 client.?.objecter ms_handle_connect 0x7f4910029ed0 2024-03-29T11:45:10.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:10.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627862915 0 0) 0x7f4910161900 con 0x7f491002b0c0 2024-03-29T11:45:10.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:10.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:10.466 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.457+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48fc004eb0 con 0x7f491002b0c0 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.458+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627872552 0 0) 0x7f4910160a80 con 0x7f4910029ed0 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.458+0000 7f490affd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.458+0000 7f490affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.458+0000 7f490affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.458+0000 7f490affd640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.458+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4910161900 con 0x7f4910029ed0 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3164534978 0 0) 0x7f4910161900 con 0x7f4910029ed0 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4910160a80 con 0x7f4910029ed0 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4123440947 0 0) 0x7f48fc004eb0 con 0x7f491002b0c0 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4910161900 con 0x7f491002b0c0 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f490c0018d0 con 0x7f4910029ed0 2024-03-29T11:45:10.467 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient(hunting): dump: 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:10.468 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4914001990 con 0x7f491002b0c0 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient(hunting): dump: 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:10.469 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:10.470 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:10.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4116775133 0 0) 0x7f4910161900 con 0x7f491002b0c0 2024-03-29T11:45:10.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 >> v1:172.21.15.203:6789/0 conn(0x7f49100c9590 legacy=0x7f491016c360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:45:10.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 >> v1:172.21.15.145:6789/0 conn(0x7f4910029ed0 legacy=0x7f49101611f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:10.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:10.471 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 monclient: found mon.c 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100c1ea0 con 0x7f491002b0c0 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient: _finish_auth 0 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f490affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f49207ac640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f49207ac640 10 monclient: _renew_subs 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f49207ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491002d250 con 0x7f491002b0c0 2024-03-29T11:45:10.472 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f49207ac640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f49207ac640 10 monclient: _renew_subs 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f49207ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49100ccae0 con 0x7f491002b0c0 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.459+0000 7f49207ac640 1 librados: init done 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.460+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4914001af0 con 0x7f491002b0c0 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.460+0000 7f490affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.460+0000 7f491cd45640 4 set_mon_vals no callback set 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.461+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f4914003a60 con 0x7f491002b0c0 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.461+0000 7f490affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.461+0000 7f490affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:10.473 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.461+0000 7f490affd640 10 monclient: dump: 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.463+0000 7f49207ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:10.474 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.463+0000 7f49207ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:10.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.463+0000 7f49207ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:10.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.463+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4900005080 con 0x7f491002b0c0 2024-03-29T11:45:10.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.465+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f4914003bc0 con 0x7f491002b0c0 2024-03-29T11:45:10.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.465+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 809964202 0 0) 0x7f4914084a50 con 0x7f491002b0c0 2024-03-29T11:45:10.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.466+0000 7f490affd640 10 client.4622.objecter ms_dispatch 0x7f4918000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:10.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.466+0000 7f490affd640 3 client.4622.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:10.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.466+0000 7f490affd640 3 client.4622.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:10.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.466+0000 7f491e548640 10 monclient: get_auth_request con 0x7f48fc06df70 auth_method 0 2024-03-29T11:45:10.475 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.466+0000 7f490affd640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:45:10.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.466+0000 7f490affd640 10 client.4622.objecter ms_handle_connect 0x7f48fc06df70 2024-03-29T11:45:10.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.474+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4914050160 con 0x7f491002b0c0 2024-03-29T11:45:10.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.474+0000 7f490affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:10.479 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.474+0000 7f490affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:10.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.800+0000 7f49207ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:10.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.800+0000 7f49207ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:10.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.800+0000 7f49207ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:10.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.800+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4900005330 con 0x7f491002b0c0 2024-03-29T11:45:10.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.801+0000 7f490affd640 1 -- 172.21.15.145:0/1731326872 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3238828718) 0x7f4914052260 con 0x7f491002b0c0 2024-03-29T11:45:10.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.801+0000 7f490affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:10.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.801+0000 7f490affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:10.806 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:45:10.806 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":11,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:45:09.876003+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:09.876003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:10.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 10 librados: watch_flush enter 2024-03-29T11:45:10.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 10 librados: watch_flush exit 2024-03-29T11:45:10.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:45:10.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:45:10.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 >> v1:172.21.15.145:6816/227372239 conn(0x7f48fc06df70 legacy=0x7f48fc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:10.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 10 monclient: shutdown 2024-03-29T11:45:10.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:10.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 >> v1:172.21.15.145:6790/0 conn(0x7f491002b0c0 legacy=0x7f49100c1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:10.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 shutdown_connections 2024-03-29T11:45:10.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.805+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 >> 172.21.15.145:0/1731326872 conn(0x7f4910005910 msgr2=0x7f49100384a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:10.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.808+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 shutdown_connections 2024-03-29T11:45:10.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.808+0000 7f49207ac640 1 -- 172.21.15.145:0/1731326872 wait complete. 2024-03-29T11:45:10.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:10.808+0000 7f49207ac640 1 librados: shutdown 2024-03-29T11:45:10.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:17.639 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:17.785 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.780+0000 7f960cd09640 10 monclient: get_monmap_and_config 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.780+0000 7f960cd09640 10 monclient: build_initial_monmap 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.781+0000 7f960cd09640 10 monclient: monmap: 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:45:17.782275+0000 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:45:17.782275+0000 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:45:17.786 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:45:17.787 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:17.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 1 Processor -- start 2024-03-29T11:45:17.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 1 -- start start 2024-03-29T11:45:17.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 10 monclient: init 2024-03-29T11:45:17.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:17.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:17.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 10 monclient(hunting): picked mon.noname-c con 0x7f960806d7e0 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:17.787 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 10 monclient(hunting): picked mon.noname-a con 0x7f9608072f20 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:17.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 10 monclient(hunting): picked mon.noname-b con 0x7f96080a9e90 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:17.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96080ad8e0 con 0x7f9608072f20 2024-03-29T11:45:17.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96080ac6a0 con 0x7f960806d7e0 2024-03-29T11:45:17.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96080ab230 con 0x7f96080a9e90 2024-03-29T11:45:17.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:17.788 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.782+0000 7f960cd09640 10 monclient(hunting): authenticate will time out at 1507.090973s 2024-03-29T11:45:17.790 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.786+0000 7f9606ffd640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f9608072f20 0x7f96080acac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:48545/0 (socket says 172.21.15.145:48545) 2024-03-29T11:45:17.791 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.786+0000 7f9606ffd640 1 -- 172.21.15.145:0/495781631 learned_addr learned my addr 172.21.15.145:0/495781631 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:17.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 -- 172.21.15.145:0/495781631 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924935800 0 0) 0x7f96080ab230 con 0x7f96080a9e90 2024-03-29T11:45:17.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:17.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:17.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:17.794 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:45:17.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 -- 172.21.15.145:0/495781631 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f0003520 con 0x7f96080a9e90 2024-03-29T11:45:17.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 -- 172.21.15.145:0/495781631 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 408831042 0 0) 0x7f95f0003520 con 0x7f96080a9e90 2024-03-29T11:45:17.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 -- 172.21.15.145:0/495781631 >> v1:172.21.15.203:6789/0 conn(0x7f960806d7e0 legacy=0x7f960813af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 -- 172.21.15.145:0/495781631 >> v1:172.21.15.145:6789/0 conn(0x7f9608072f20 legacy=0x7f96080acac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:17.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 monclient: found mon.noname-b 2024-03-29T11:45:17.795 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 -- 172.21.15.145:0/495781631 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96080aaba0 con 0x7f96080a9e90 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 -- 172.21.15.145:0/495781631 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f95f8001690 con 0x7f96080a9e90 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: dump: 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:17.796 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 -- 172.21.15.145:0/495781631 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95f80018b0 con 0x7f96080a9e90 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 1 -- 172.21.15.145:0/495781631 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f95f8001c40 con 0x7f96080a9e90 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.790+0000 7f96067fc640 10 monclient: dump: 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:17.797 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:17.798 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:17.798 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:17.798 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:17.798 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:17.798 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:17.798 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.791+0000 7f960cd09640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.791+0000 7f960cd09640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.791+0000 7f960cd09640 4 set_mon_vals no callback set 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.791+0000 7f960cd09640 10 monclient: shutdown 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.791+0000 7f960cd09640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.791+0000 7f960cd09640 1 -- 172.21.15.145:0/495781631 >> v1:172.21.15.145:6790/0 conn(0x7f96080a9e90 legacy=0x7f96080ad1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.792+0000 7f960cd09640 1 -- 172.21.15.145:0/495781631 shutdown_connections 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.792+0000 7f960cd09640 1 -- 172.21.15.145:0/495781631 >> 172.21.15.145:0/495781631 conn(0x7f96080b3a30 msgr2=0x7f96080b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.792+0000 7f960cd09640 1 -- 172.21.15.145:0/495781631 shutdown_connections 2024-03-29T11:45:17.799 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.792+0000 7f960cd09640 1 -- 172.21.15.145:0/495781631 wait complete. 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.793+0000 7f960cd09640 10 monclient: build_initial_monmap 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.793+0000 7f960cd09640 10 monclient: monmap: 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:17.804 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.793+0000 7f960cd09640 1 librados: starting msgr at 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.793+0000 7f960cd09640 1 librados: starting objecter 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.793+0000 7f960cd09640 1 Processor -- start 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 1 -- start start 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 1 librados: setting wanted keys 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 1 librados: calling monclient init 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 10 monclient: init 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 10 monclient(hunting): picked mon.noname-b con 0x7f960806d7e0 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 10 monclient(hunting): picked mon.noname-c con 0x7f9608072f20 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 10 monclient(hunting): picked mon.noname-a con 0x7f96080a9e90 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:17.805 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96080ab000 con 0x7f96080a9e90 2024-03-29T11:45:17.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f960813e2c0 con 0x7f960806d7e0 2024-03-29T11:45:17.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9608151420 con 0x7f9608072f20 2024-03-29T11:45:17.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:17.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.794+0000 7f960cd09640 10 monclient(hunting): authenticate will time out at 1507.102843s 2024-03-29T11:45:17.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.799+0000 7f9606ffd640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f9608072f20 0x7f9608151e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:37797/0 (socket says 172.21.15.145:37797) 2024-03-29T11:45:17.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.799+0000 7f9606ffd640 1 -- 172.21.15.145:0/3117415180 learned_addr learned my addr 172.21.15.145:0/3117415180 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:17.806 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.800+0000 7f95effff640 10 client.?.objecter ms_handle_connect 0x7f9608072f20 2024-03-29T11:45:17.807 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.800+0000 7f95effff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:17.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 client.?.objecter ms_handle_connect 0x7f96080a9e90 2024-03-29T11:45:17.809 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842817003 0 0) 0x7f96080ab000 con 0x7f96080a9e90 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f4004eb0 con 0x7f96080a9e90 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113218836 0 0) 0x7f9608151420 con 0x7f9608072f20 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96080ab000 con 0x7f9608072f20 2024-03-29T11:45:17.810 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2122478390 0 0) 0x7f96080ab000 con 0x7f9608072f20 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9608151420 con 0x7f9608072f20 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9600001b60 con 0x7f9608072f20 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): dump: 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:17.811 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.804+0000 7f95effff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 361882852 0 0) 0x7f9608151420 con 0x7f9608072f20 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 >> v1:172.21.15.203:6789/0 conn(0x7f960806d7e0 legacy=0x7f96081e3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 >> v1:172.21.15.145:6789/0 conn(0x7f96080a9e90 legacy=0x7f9608152510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 1 monclient: found mon.c 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9608151700 con 0x7f9608072f20 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 10 monclient: _finish_auth 0 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f96000020d0 con 0x7f9608072f20 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:17.812 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f9600004d00 con 0x7f9608072f20 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.805+0000 7f95effff640 10 monclient: dump: 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:17.813 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:17.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f96057fa640 4 set_mon_vals no callback set 2024-03-29T11:45:17.814 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f960cd09640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:45:17.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f960cd09640 10 monclient: _renew_subs 2024-03-29T11:45:17.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f960cd09640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:17.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960813de50 con 0x7f9608072f20 2024-03-29T11:45:17.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f960cd09640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:17.815 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f960cd09640 10 monclient: _renew_subs 2024-03-29T11:45:17.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f960cd09640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:17.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96081e3a10 con 0x7f9608072f20 2024-03-29T11:45:17.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.806+0000 7f960cd09640 1 librados: init done 2024-03-29T11:45:17.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.808+0000 7f960cd09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:17.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.808+0000 7f960cd09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:17.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.808+0000 7f960cd09640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:17.816 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.808+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960813e090 con 0x7f9608072f20 2024-03-29T11:45:17.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.812+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f9600004e60 con 0x7f9608072f20 2024-03-29T11:45:17.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.812+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 809964202 0 0) 0x7f9600054b00 con 0x7f9608072f20 2024-03-29T11:45:17.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.812+0000 7f95effff640 10 client.4628.objecter ms_dispatch 0x7f9608000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:17.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.812+0000 7f95effff640 3 client.4628.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:17.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.812+0000 7f95effff640 3 client.4628.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:17.818 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.812+0000 7f96077fe640 10 monclient: get_auth_request con 0x7f95f406dbf0 auth_method 0 2024-03-29T11:45:17.819 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.812+0000 7f95effff640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:45:17.819 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.813+0000 7f95effff640 10 client.4628.objecter ms_handle_connect 0x7f95f406dbf0 2024-03-29T11:45:17.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.821+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9600050da0 con 0x7f9608072f20 2024-03-29T11:45:17.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.821+0000 7f95effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:17.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:17.821+0000 7f95effff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:18.149 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.145+0000 7f960cd09640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:18.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.145+0000 7f960cd09640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:18.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.145+0000 7f960cd09640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:18.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.145+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f960813dfe0 con 0x7f9608072f20 2024-03-29T11:45:18.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.145+0000 7f95effff640 1 -- 172.21.15.145:0/3117415180 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3238828718) 0x7f9600050da0 con 0x7f9608072f20 2024-03-29T11:45:18.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.145+0000 7f95effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:18.150 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.145+0000 7f95effff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:18.151 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:45:18.151 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":11,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:45:09.876003+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:09.876003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:18.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.149+0000 7f960cd09640 10 librados: watch_flush enter 2024-03-29T11:45:18.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.149+0000 7f960cd09640 10 librados: watch_flush exit 2024-03-29T11:45:18.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.149+0000 7f960cd09640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:45:18.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.149+0000 7f960cd09640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:45:18.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.149+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 >> v1:172.21.15.145:6816/227372239 conn(0x7f95f406dbf0 legacy=0x7f95f40700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:18.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.150+0000 7f960cd09640 10 monclient: shutdown 2024-03-29T11:45:18.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.150+0000 7f960cd09640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:18.155 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.150+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 >> v1:172.21.15.145:6790/0 conn(0x7f9608072f20 legacy=0x7f9608151e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:18.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.151+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 shutdown_connections 2024-03-29T11:45:18.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.151+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 >> 172.21.15.145:0/3117415180 conn(0x7f96080b3a30 msgr2=0x7f9608137f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:18.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.152+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 shutdown_connections 2024-03-29T11:45:18.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.152+0000 7f960cd09640 1 -- 172.21.15.145:0/3117415180 wait complete. 2024-03-29T11:45:18.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:18.152+0000 7f960cd09640 1 librados: shutdown 2024-03-29T11:45:18.174 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:24.982 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:25.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.120+0000 7f7480d09640 10 monclient: get_monmap_and_config 2024-03-29T11:45:25.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.120+0000 7f7480d09640 10 monclient: build_initial_monmap 2024-03-29T11:45:25.128 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.120+0000 7f7480d09640 10 monclient: monmap: 2024-03-29T11:45:25.128 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:45:25.121871+0000 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:45:25.121871+0000 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 1 Processor -- start 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 1 -- start start 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 10 monclient: init 2024-03-29T11:45:25.129 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:25.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:25.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 10 monclient(hunting): picked mon.noname-b con 0x7f7474029f00 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:25.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 10 monclient(hunting): picked mon.noname-a con 0x7f747402b090 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:25.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 10 monclient(hunting): picked mon.noname-c con 0x7f74740e2c40 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:25.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747402c0f0 con 0x7f747402b090 2024-03-29T11:45:25.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747402a430 con 0x7f74740e2c40 2024-03-29T11:45:25.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747402a5c0 con 0x7f7474029f00 2024-03-29T11:45:25.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.122+0000 7f7480d09640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:25.130 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.123+0000 7f7480d09640 10 monclient(hunting): authenticate will time out at 1514.431028s 2024-03-29T11:45:25.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.125+0000 7f747bfff640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f74740e2c40 0x7f74740f0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:51320/0 (socket says 172.21.15.145:51320) 2024-03-29T11:45:25.131 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.125+0000 7f747bfff640 1 -- 172.21.15.145:0/2100601108 learned_addr learned my addr 172.21.15.145:0/2100601108 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:25.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.126+0000 7f747a7fc640 1 -- 172.21.15.145:0/2100601108 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965066903 0 0) 0x7f747402a430 con 0x7f74740e2c40 2024-03-29T11:45:25.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.126+0000 7f747a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:25.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.126+0000 7f747a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:25.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.126+0000 7f747a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:25.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.126+0000 7f747a7fc640 10 monclient(hunting): my global_id is 4591 2024-03-29T11:45:25.132 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.126+0000 7f747a7fc640 1 -- 172.21.15.145:0/2100601108 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f746c003520 con 0x7f74740e2c40 2024-03-29T11:45:25.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 1 -- 172.21.15.145:0/2100601108 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 922637080 0 0) 0x7f746c003520 con 0x7f74740e2c40 2024-03-29T11:45:25.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 1 -- 172.21.15.145:0/2100601108 >> v1:172.21.15.145:6790/0 conn(0x7f7474029f00 legacy=0x7f74740e8900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:45:25.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 1 -- 172.21.15.145:0/2100601108 >> v1:172.21.15.145:6789/0 conn(0x7f747402b090 legacy=0x7f74740e2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:25.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:25.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:45:25.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:25.133 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.203:6789/0 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 1 -- 172.21.15.145:0/2100601108 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7474006940 con 0x7f74740e2c40 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 1 -- 172.21.15.145:0/2100601108 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f74680012f0 con 0x7f74740e2c40 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.127+0000 7f747a7fc640 10 monclient: dump: 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.134 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:25.135 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:25.135 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:25.135 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:25.135 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:25.135 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:25.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f7480d09640 5 monclient: authenticate success, global_id 4591 2024-03-29T11:45:25.135 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f7480d09640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:25.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f747a7fc640 1 -- 172.21.15.145:0/2100601108 <== mon.1 v1:172.21.15.203:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74680014c0 con 0x7f74740e2c40 2024-03-29T11:45:25.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f747a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:25.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f747a7fc640 1 -- 172.21.15.145:0/2100601108 <== mon.1 v1:172.21.15.203:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f74680018d0 con 0x7f74740e2c40 2024-03-29T11:45:25.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f7480d09640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:25.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f7480d09640 4 set_mon_vals no callback set 2024-03-29T11:45:25.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f747a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:25.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f747a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:25.137 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.129+0000 7f747a7fc640 10 monclient: dump: 2024-03-29T11:45:25.137 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.130+0000 7f7480d09640 10 monclient: shutdown 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.130+0000 7f7480d09640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.130+0000 7f7480d09640 1 -- 172.21.15.145:0/2100601108 >> v1:172.21.15.203:6789/0 conn(0x7f74740e2c40 legacy=0x7f74740f0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.130+0000 7f7480d09640 1 -- 172.21.15.145:0/2100601108 shutdown_connections 2024-03-29T11:45:25.138 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.130+0000 7f7480d09640 1 -- 172.21.15.145:0/2100601108 >> 172.21.15.145:0/2100601108 conn(0x7f74740058f0 msgr2=0x7f7474005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:25.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.131+0000 7f7480d09640 1 -- 172.21.15.145:0/2100601108 shutdown_connections 2024-03-29T11:45:25.156 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.133+0000 7f7480d09640 1 -- 172.21.15.145:0/2100601108 wait complete. 2024-03-29T11:45:25.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.134+0000 7f7480d09640 10 monclient: build_initial_monmap 2024-03-29T11:45:25.157 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.134+0000 7f7480d09640 10 monclient: monmap: 2024-03-29T11:45:25.157 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:25.157 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:25.158 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:45:25.158 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:45:25.158 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:25.158 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:25.158 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:25.158 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:45:25.158 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:45:25.158 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:25.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.134+0000 7f7480d09640 1 librados: starting msgr at 2024-03-29T11:45:25.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.134+0000 7f7480d09640 1 librados: starting objecter 2024-03-29T11:45:25.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.134+0000 7f7480d09640 1 Processor -- start 2024-03-29T11:45:25.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.135+0000 7f7480d09640 1 -- start start 2024-03-29T11:45:25.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.135+0000 7f7480d09640 1 librados: setting wanted keys 2024-03-29T11:45:25.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.135+0000 7f7480d09640 1 librados: calling monclient init 2024-03-29T11:45:25.159 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.135+0000 7f7480d09640 10 monclient: init 2024-03-29T11:45:25.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.135+0000 7f7480d09640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:25.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.135+0000 7f7480d09640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:45:25.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.135+0000 7f7480d09640 10 monclient(hunting): picked mon.noname-b con 0x7f7474029f00 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:25.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.136+0000 7f7480d09640 10 monclient(hunting): picked mon.noname-a con 0x7f747402b090 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:25.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.136+0000 7f7480d09640 10 monclient(hunting): picked mon.noname-c con 0x7f74740e2c40 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:25.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.136+0000 7f7480d09640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74740d9dd0 con 0x7f747402b090 2024-03-29T11:45:25.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.136+0000 7f7480d09640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74740dfb40 con 0x7f7474029f00 2024-03-29T11:45:25.160 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.136+0000 7f7480d09640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747402e480 con 0x7f74740e2c40 2024-03-29T11:45:25.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.136+0000 7f7480d09640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:25.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.136+0000 7f7480d09640 10 monclient(hunting): authenticate will time out at 1514.444081s 2024-03-29T11:45:25.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.138+0000 7f747b7fe640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f7474029f00 0x7f74740322e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:52264/0 (socket says 172.21.15.145:52264) 2024-03-29T11:45:25.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.138+0000 7f747b7fe640 1 -- 172.21.15.145:0/1603671689 learned_addr learned my addr 172.21.15.145:0/1603671689 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:25.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.138+0000 7f7467fff640 10 client.?.objecter ms_handle_connect 0x7f747402b090 2024-03-29T11:45:25.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.138+0000 7f7467fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:25.161 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.138+0000 7f7467fff640 10 client.?.objecter ms_handle_connect 0x7f7474029f00 2024-03-29T11:45:25.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.138+0000 7f7467fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:25.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052512350 0 0) 0x7f74740d9dd0 con 0x7f747402b090 2024-03-29T11:45:25.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:25.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:25.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:25.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): my global_id is 4602 2024-03-29T11:45:25.162 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460004eb0 con 0x7f747402b090 2024-03-29T11:45:25.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167990430 0 0) 0x7f74740dfb40 con 0x7f7474029f00 2024-03-29T11:45:25.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:25.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:25.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:25.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): my global_id is 4594 2024-03-29T11:45:25.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7460006230 con 0x7f7474029f00 2024-03-29T11:45:25.163 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1746491133 0 0) 0x7f7460004eb0 con 0x7f747402b090 2024-03-29T11:45:25.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74740dfb40 con 0x7f747402b090 2024-03-29T11:45:25.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f747c04fdd0 con 0x7f747402b090 2024-03-29T11:45:25.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:25.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:25.164 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): dump: 2024-03-29T11:45:25.164 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:25.165 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:25.165 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.165 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.165 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:25.165 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:25.165 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:25.166 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:25.166 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:25.166 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:25.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:25.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2407536334 0 0) 0x7f7460006230 con 0x7f7474029f00 2024-03-29T11:45:25.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7460006ba0 con 0x7f7474029f00 2024-03-29T11:45:25.166 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f7470001960 con 0x7f7474029f00 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 10 monclient(hunting): dump: 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:25.167 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:25.168 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:25.168 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:25.168 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:25.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.139+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2887744663 0 0) 0x7f7460006ba0 con 0x7f7474029f00 2024-03-29T11:45:25.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 >> v1:172.21.15.145:6790/0 conn(0x7f74740e2c40 legacy=0x7f74740ddb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:25.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 >> v1:172.21.15.145:6789/0 conn(0x7f747402b090 legacy=0x7f74740dd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:25.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:25.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 1 monclient: found mon.b 2024-03-29T11:45:25.168 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:25.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:25.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747402e6f0 con 0x7f7474029f00 2024-03-29T11:45:25.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:25.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:25.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7467fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:25.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7480d09640 5 monclient: authenticate success, global_id 4594 2024-03-29T11:45:25.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7480d09640 10 monclient: _renew_subs 2024-03-29T11:45:25.169 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7480d09640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:25.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747402e9d0 con 0x7f7474029f00 2024-03-29T11:45:25.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7480d09640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:25.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7480d09640 10 monclient: _renew_subs 2024-03-29T11:45:25.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7480d09640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:25.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f747417e810 con 0x7f7474029f00 2024-03-29T11:45:25.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.140+0000 7f7480d09640 1 librados: init done 2024-03-29T11:45:25.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.142+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74700035f0 con 0x7f7474029f00 2024-03-29T11:45:25.170 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.142+0000 7f7467fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:25.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.142+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f7470003940 con 0x7f7474029f00 2024-03-29T11:45:25.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.142+0000 7f7467fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:25.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.142+0000 7f7467fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:25.171 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.142+0000 7f7467fff640 10 monclient: dump: 2024-03-29T11:45:25.171 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:25.171 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:25.172 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.172 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:25.172 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:25.172 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:25.172 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:25.172 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:25.172 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:25.173 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:25.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.142+0000 7f7479ffb640 4 set_mon_vals no callback set 2024-03-29T11:45:25.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.142+0000 7f7480d09640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:25.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.143+0000 7f7480d09640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:25.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.143+0000 7f7480d09640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:25.173 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.143+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f745c005080 con 0x7f7474029f00 2024-03-29T11:45:25.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.144+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f7470003aa0 con 0x7f7474029f00 2024-03-29T11:45:25.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.145+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 809964202 0 0) 0x7f7470084a60 con 0x7f7474029f00 2024-03-29T11:45:25.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.145+0000 7f7467fff640 10 client.4594.objecter ms_dispatch 0x7f747c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:25.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.145+0000 7f7467fff640 3 client.4594.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:25.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.145+0000 7f7467fff640 3 client.4594.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:25.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.145+0000 7f7467fff640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-29T11:45:25.174 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.145+0000 7f747affd640 10 monclient: get_auth_request con 0x7f746006e280 auth_method 0 2024-03-29T11:45:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.145+0000 7f7467fff640 10 client.4594.objecter ms_handle_connect 0x7f746006e280 2024-03-29T11:45:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.152+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7470050340 con 0x7f7474029f00 2024-03-29T11:45:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.152+0000 7f7467fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:25.175 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.152+0000 7f7467fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:25.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.476+0000 7f7480d09640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:25.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.476+0000 7f7480d09640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:25.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.476+0000 7f7480d09640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:25.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.476+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f745c005330 con 0x7f7474029f00 2024-03-29T11:45:25.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.477+0000 7f7467fff640 1 -- 172.21.15.145:0/1603671689 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3238828718) 0x7f7470052270 con 0x7f7474029f00 2024-03-29T11:45:25.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.477+0000 7f7467fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:25.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.477+0000 7f7467fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:25.484 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:45:25.485 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":11,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:45:09.876003+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:09.876003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:25.488 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.483+0000 7f7480d09640 10 librados: watch_flush enter 2024-03-29T11:45:25.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.484+0000 7f7480d09640 10 librados: watch_flush exit 2024-03-29T11:45:25.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.484+0000 7f7480d09640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-29T11:45:25.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.484+0000 7f7480d09640 10 client.4594.objecter successfully canceled tick 2024-03-29T11:45:25.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.484+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 >> v1:172.21.15.145:6816/227372239 conn(0x7f746006e280 legacy=0x7f7460070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:25.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.484+0000 7f7480d09640 10 monclient: shutdown 2024-03-29T11:45:25.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.484+0000 7f7480d09640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:25.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.484+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 >> v1:172.21.15.203:6789/0 conn(0x7f7474029f00 legacy=0x7f74740322e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:25.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.484+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 shutdown_connections 2024-03-29T11:45:25.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.484+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 >> 172.21.15.145:0/1603671689 conn(0x7f74740058f0 msgr2=0x7f74740e2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:25.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.485+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 shutdown_connections 2024-03-29T11:45:25.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.485+0000 7f7480d09640 1 -- 172.21.15.145:0/1603671689 wait complete. 2024-03-29T11:45:25.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:25.485+0000 7f7480d09640 1 librados: shutdown 2024-03-29T11:45:25.507 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:32.314 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:32.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.469+0000 7f63c2b32640 10 monclient: get_monmap_and_config 2024-03-29T11:45:32.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.469+0000 7f63c2b32640 10 monclient: build_initial_monmap 2024-03-29T11:45:32.476 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.469+0000 7f63c2b32640 10 monclient: monmap: 2024-03-29T11:45:32.476 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:32.476 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:32.476 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:45:32.471134+0000 2024-03-29T11:45:32.476 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:45:32.471134+0000 2024-03-29T11:45:32.477 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:32.477 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:32.477 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:32.477 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:45:32.477 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:45:32.477 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:32.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.471+0000 7f63c2b32640 1 Processor -- start 2024-03-29T11:45:32.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.471+0000 7f63c2b32640 1 -- start start 2024-03-29T11:45:32.477 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.471+0000 7f63c2b32640 10 monclient: init 2024-03-29T11:45:32.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:32.480 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 10 monclient(hunting): picked mon.noname-c con 0x7f63bc0a9750 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 10 monclient(hunting): picked mon.noname-b con 0x7f63bc149ac0 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 10 monclient(hunting): picked mon.noname-a con 0x7f63bc14fe50 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc0aa4a0 con 0x7f63bc14fe50 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc14a680 con 0x7f63bc0a9750 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc0af290 con 0x7f63bc149ac0 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c2b32640 10 monclient(hunting): authenticate will time out at 1521.780342s 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c08ce640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f63bc0a9750 0x7f63bc1436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:34900/0 (socket says 172.21.15.145:34900) 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.472+0000 7f63c08ce640 1 -- 172.21.15.145:0/1872049430 learned_addr learned my addr 172.21.15.145:0/1872049430 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 1 -- 172.21.15.145:0/1872049430 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111519794 0 0) 0x7f63bc0af290 con 0x7f63bc149ac0 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:32.481 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 1 -- 172.21.15.145:0/1872049430 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac003520 con 0x7f63bc149ac0 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 1 -- 172.21.15.145:0/1872049430 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178195709 0 0) 0x7f63ac003520 con 0x7f63bc149ac0 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 1 -- 172.21.15.145:0/1872049430 >> v1:172.21.15.203:6789/0 conn(0x7f63bc0a9750 legacy=0x7f63bc1436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 1 -- 172.21.15.145:0/1872049430 >> v1:172.21.15.145:6789/0 conn(0x7f63bc14fe50 legacy=0x7f63bc152240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 1 monclient: found mon.noname-b 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 1 -- 172.21.15.145:0/1872049430 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0afb40 con 0x7f63bc149ac0 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:32.482 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63c2b32640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63c2b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 1 -- 172.21.15.145:0/1872049430 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f63a8001690 con 0x7f63bc149ac0 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63bb7fe640 10 monclient: dump: 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:32.483 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.473+0000 7f63c2b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.474+0000 7f63bb7fe640 1 -- 172.21.15.145:0/1872049430 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63a80018b0 con 0x7f63bc149ac0 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.474+0000 7f63bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.474+0000 7f63c2b32640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.474+0000 7f63c2b32640 4 set_mon_vals no callback set 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.474+0000 7f63c2b32640 10 monclient: shutdown 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.474+0000 7f63c2b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.474+0000 7f63c2b32640 1 -- 172.21.15.145:0/1872049430 >> v1:172.21.15.145:6790/0 conn(0x7f63bc149ac0 legacy=0x7f63bc149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.475+0000 7f63c2b32640 1 -- 172.21.15.145:0/1872049430 shutdown_connections 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.475+0000 7f63c2b32640 1 -- 172.21.15.145:0/1872049430 >> 172.21.15.145:0/1872049430 conn(0x7f63bc0b35e0 msgr2=0x7f63bc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.475+0000 7f63c2b32640 1 -- 172.21.15.145:0/1872049430 shutdown_connections 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.475+0000 7f63c2b32640 1 -- 172.21.15.145:0/1872049430 wait complete. 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.476+0000 7f63c2b32640 10 monclient: build_initial_monmap 2024-03-29T11:45:32.484 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.476+0000 7f63c2b32640 10 monclient: monmap: 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:45:32.485 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:32.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.476+0000 7f63c2b32640 1 librados: starting msgr at 2024-03-29T11:45:32.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.476+0000 7f63c2b32640 1 librados: starting objecter 2024-03-29T11:45:32.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.476+0000 7f63c2b32640 1 Processor -- start 2024-03-29T11:45:32.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.477+0000 7f63c2b32640 1 -- start start 2024-03-29T11:45:32.489 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.477+0000 7f63c2b32640 1 librados: setting wanted keys 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.477+0000 7f63c2b32640 1 librados: calling monclient init 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.477+0000 7f63c2b32640 10 monclient: init 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 10 monclient(hunting): picked mon.noname-c con 0x7f63bc0a9750 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 10 monclient(hunting): picked mon.noname-a con 0x7f63bc149ac0 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 10 monclient(hunting): picked mon.noname-b con 0x7f63bc14fe50 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc11def0 con 0x7f63bc149ac0 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc11d080 con 0x7f63bc14fe50 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc11e4d0 con 0x7f63bc0a9750 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c2b32640 10 monclient(hunting): authenticate will time out at 1521.786282s 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c08ce640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f63bc0a9750 0x7f63bc140570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:35904/0 (socket says 172.21.15.145:35904) 2024-03-29T11:45:32.490 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63c08ce640 1 -- 172.21.15.145:0/304306584 learned_addr learned my addr 172.21.15.145:0/304306584 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63b97fa640 10 client.?.objecter ms_handle_connect 0x7f63bc0a9750 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63b97fa640 10 client.?.objecter ms_handle_connect 0x7f63bc149ac0 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63b97fa640 10 client.?.objecter ms_handle_connect 0x7f63bc14fe50 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.478+0000 7f63b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154295864 0 0) 0x7f63bc11d080 con 0x7f63bc14fe50 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 10 monclient(hunting): my global_id is 4600 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a0004eb0 con 0x7f63bc14fe50 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287409409 0 0) 0x7f63bc11e4d0 con 0x7f63bc0a9750 2024-03-29T11:45:32.491 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.479+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63bc11d080 con 0x7f63bc0a9750 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224237583 0 0) 0x7f63bc11def0 con 0x7f63bc149ac0 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): my global_id is 4608 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63bc11e4d0 con 0x7f63bc149ac0 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 131142736 0 0) 0x7f63a0004eb0 con 0x7f63bc14fe50 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63bc11def0 con 0x7f63bc14fe50 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f63b4001ae0 con 0x7f63bc14fe50 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:32.492 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): dump: 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4027048090 0 0) 0x7f63bc11d080 con 0x7f63bc0a9750 2024-03-29T11:45:32.493 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63a0004eb0 con 0x7f63bc0a9750 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f63b0001850 con 0x7f63bc0a9750 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 10 monclient(hunting): dump: 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:32.494 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3990767654 0 0) 0x7f63bc11def0 con 0x7f63bc14fe50 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.480+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 >> v1:172.21.15.145:6790/0 conn(0x7f63bc0a9750 legacy=0x7f63bc140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 >> v1:172.21.15.145:6789/0 conn(0x7f63bc149ac0 legacy=0x7f63bc1dee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 1 monclient: found mon.b 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.203:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc11e660 con 0x7f63bc14fe50 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 10 monclient: _finish_auth 0 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63c2b32640 5 monclient: authenticate success, global_id 4600 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63c2b32640 10 monclient: _renew_subs 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63c2b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:32.495 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.203:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc11d8f0 con 0x7f63bc14fe50 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63c2b32640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63c2b32640 10 monclient: _renew_subs 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63c2b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.203:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc071ff0 con 0x7f63bc14fe50 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63c2b32640 1 librados: init done 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63b4001c40 con 0x7f63bc14fe50 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.481+0000 7f63baffd640 4 set_mon_vals no callback set 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.482+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f63b4003970 con 0x7f63bc14fe50 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.482+0000 7f63b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.482+0000 7f63b97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.482+0000 7f63b97fa640 10 monclient: dump: 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:32.496 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.482+0000 7f63c2b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.483+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f63b4003ad0 con 0x7f63bc14fe50 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.484+0000 7f63c2b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.484+0000 7f63c2b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:32.497 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.484+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639c005080 con 0x7f63bc14fe50 2024-03-29T11:45:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.485+0000 7f63c08ce640 10 monclient: get_auth_request con 0x7f63a006df90 auth_method 0 2024-03-29T11:45:32.498 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.486+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 809964202 0 0) 0x7f63b40839c0 con 0x7f63bc14fe50 2024-03-29T11:45:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.486+0000 7f63b97fa640 10 client.4600.objecter ms_dispatch 0x7f63bc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.486+0000 7f63b97fa640 3 client.4600.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.486+0000 7f63b97fa640 3 client.4600.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.486+0000 7f63b97fa640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-29T11:45:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.486+0000 7f63b97fa640 10 client.4600.objecter ms_handle_connect 0x7f63a006df90 2024-03-29T11:45:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.492+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f63b404f2a0 con 0x7f63bc14fe50 2024-03-29T11:45:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.492+0000 7f63b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:32.499 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.492+0000 7f63b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:32.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.880+0000 7f63c2b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.880+0000 7f63c2b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:32.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.880+0000 7f63c2b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.203:6789/0 2024-03-29T11:45:32.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.880+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 --> v1:172.21.15.203:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f639c005330 con 0x7f63bc14fe50 2024-03-29T11:45:32.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.881+0000 7f63b97fa640 1 -- 172.21.15.145:0/304306584 <== mon.1 v1:172.21.15.203:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3238828718) 0x7f63b40511d0 con 0x7f63bc14fe50 2024-03-29T11:45:32.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.881+0000 7f63b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:32.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.881+0000 7f63b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:32.887 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:45:32.887 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":11,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:45:09.876003+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:09.876003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:32.889 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.885+0000 7f63c2b32640 10 librados: watch_flush enter 2024-03-29T11:45:32.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.885+0000 7f63c2b32640 10 librados: watch_flush exit 2024-03-29T11:45:32.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.885+0000 7f63c2b32640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-29T11:45:32.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.885+0000 7f63c2b32640 10 client.4600.objecter successfully canceled tick 2024-03-29T11:45:32.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.885+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 >> v1:172.21.15.145:6816/227372239 conn(0x7f63a006df90 legacy=0x7f63a0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:32.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.885+0000 7f63c2b32640 10 monclient: shutdown 2024-03-29T11:45:32.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.886+0000 7f63c2b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:32.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.886+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 >> v1:172.21.15.203:6789/0 conn(0x7f63bc14fe50 legacy=0x7f63bc1e71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:32.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.886+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 shutdown_connections 2024-03-29T11:45:32.891 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.886+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 >> 172.21.15.145:0/304306584 conn(0x7f63bc0b35e0 msgr2=0x7f63bc143af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:32.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.886+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 shutdown_connections 2024-03-29T11:45:32.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.886+0000 7f63c2b32640 1 -- 172.21.15.145:0/304306584 wait complete. 2024-03-29T11:45:32.892 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:32.886+0000 7f63c2b32640 1 librados: shutdown 2024-03-29T11:45:32.906 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:39.714 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:39.865 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.859+0000 7f5ed1d54640 10 monclient: get_monmap_and_config 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.859+0000 7f5ed1d54640 10 monclient: build_initial_monmap 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.860+0000 7f5ed1d54640 10 monclient: monmap: 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:45:39.860267+0000 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:45:39.860267+0000 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:45:39.866 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:39.867 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.862+0000 7f5ed1d54640 1 Processor -- start 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.862+0000 7f5ed1d54640 1 -- start start 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.862+0000 7f5ed1d54640 10 monclient: init 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 10 monclient(hunting): picked mon.noname-a con 0x7f5ecc0a9ac0 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 10 monclient(hunting): picked mon.noname-b con 0x7f5ecc13ee90 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 10 monclient(hunting): picked mon.noname-c con 0x7f5ecc0af900 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc0afd10 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc0af6f0 con 0x7f5ecc0af900 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc0aac80 con 0x7f5ecc13ee90 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:39.870 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.863+0000 7f5ed1d54640 10 monclient(hunting): authenticate will time out at 1529.170744s 2024-03-29T11:45:39.871 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.865+0000 7f5ecb7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5ecc0a9ac0 0x7f5ecc0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:52842/0 (socket says 172.21.15.145:52842) 2024-03-29T11:45:39.871 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.865+0000 7f5ecb7fe640 1 -- 172.21.15.145:0/2612747622 learned_addr learned my addr 172.21.15.145:0/2612747622 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.866+0000 7f5eca7fc640 1 -- 172.21.15.145:0/2612747622 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721549755 0 0) 0x7f5ecc0afd10 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.866+0000 7f5eca7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.866+0000 7f5eca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.866+0000 7f5eca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.866+0000 7f5eca7fc640 10 monclient(hunting): my global_id is 4611 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.866+0000 7f5eca7fc640 1 -- 172.21.15.145:0/2612747622 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb4003520 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.866+0000 7f5eca7fc640 1 -- 172.21.15.145:0/2612747622 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2803036099 0 0) 0x7f5eb4003520 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.866+0000 7f5eca7fc640 1 -- 172.21.15.145:0/2612747622 >> v1:172.21.15.145:6790/0 conn(0x7f5ecc13ee90 legacy=0x7f5ecc13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.867+0000 7f5eca7fc640 1 -- 172.21.15.145:0/2612747622 >> v1:172.21.15.203:6789/0 conn(0x7f5ecc0af900 legacy=0x7f5ecc149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:39.873 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 1 monclient: found mon.noname-a 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 1 -- 172.21.15.145:0/2612747622 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc0aaf60 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: _finish_auth 0 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 1 -- 172.21.15.145:0/2612747622 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ec4001a10 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: dump: 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:39.874 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5ed1d54640 5 monclient: authenticate success, global_id 4611 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5ed1d54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 1 -- 172.21.15.145:0/2612747622 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ec4001ca0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 1 -- 172.21.15.145:0/2612747622 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ec4002030 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5eca7fc640 10 monclient: dump: 2024-03-29T11:45:39.875 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5ed1d54640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.868+0000 7f5ed1d54640 4 set_mon_vals no callback set 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.869+0000 7f5ed1d54640 10 monclient: shutdown 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.869+0000 7f5ed1d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:39.876 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.869+0000 7f5ed1d54640 1 -- 172.21.15.145:0/2612747622 >> v1:172.21.15.145:6789/0 conn(0x7f5ecc0a9ac0 legacy=0x7f5ecc0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:39.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.871+0000 7f5ed1d54640 1 -- 172.21.15.145:0/2612747622 shutdown_connections 2024-03-29T11:45:39.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.871+0000 7f5ed1d54640 1 -- 172.21.15.145:0/2612747622 >> 172.21.15.145:0/2612747622 conn(0x7f5ecc0b35e0 msgr2=0x7f5ecc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:39.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.872+0000 7f5ed1d54640 1 -- 172.21.15.145:0/2612747622 shutdown_connections 2024-03-29T11:45:39.877 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.872+0000 7f5ed1d54640 1 -- 172.21.15.145:0/2612747622 wait complete. 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.873+0000 7f5ed1d54640 10 monclient: build_initial_monmap 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.873+0000 7f5ed1d54640 10 monclient: monmap: 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.873+0000 7f5ed1d54640 1 librados: starting msgr at 2024-03-29T11:45:39.878 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.873+0000 7f5ed1d54640 1 librados: starting objecter 2024-03-29T11:45:39.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.873+0000 7f5ed1d54640 1 Processor -- start 2024-03-29T11:45:39.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.874+0000 7f5ed1d54640 1 -- start start 2024-03-29T11:45:39.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.874+0000 7f5ed1d54640 1 librados: setting wanted keys 2024-03-29T11:45:39.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.874+0000 7f5ed1d54640 1 librados: calling monclient init 2024-03-29T11:45:39.879 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.874+0000 7f5ed1d54640 10 monclient: init 2024-03-29T11:45:39.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:39.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:39.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 10 monclient(hunting): picked mon.noname-a con 0x7f5ecc0a9ac0 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:39.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 10 monclient(hunting): picked mon.noname-c con 0x7f5ecc0af900 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:39.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 10 monclient(hunting): picked mon.noname-b con 0x7f5ecc13ee90 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:39.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc1d70b0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.882 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc1d6c40 con 0x7f5ecc13ee90 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ecc1d6dd0 con 0x7f5ecc0af900 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.875+0000 7f5ed1d54640 10 monclient(hunting): authenticate will time out at 1529.182273s 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.876+0000 7f5ecb7fe640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5ecc0a9ac0 0x7f5ecc1d6530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:60134/0 (socket says 172.21.15.145:60134) 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.876+0000 7f5ecb7fe640 1 -- 172.21.15.145:0/3009777914 learned_addr learned my addr 172.21.15.145:0/3009777914 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.877+0000 7f5eb3fff640 10 client.?.objecter ms_handle_connect 0x7f5ecc0a9ac0 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.877+0000 7f5eb3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.877+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3335278875 0 0) 0x7f5ecc1d70b0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.877+0000 7f5eb3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.877+0000 7f5eb3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.877+0000 7f5eb3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.877+0000 7f5eb3fff640 10 monclient(hunting): my global_id is 4614 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.877+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5eb8004eb0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2445338239 0 0) 0x7f5eb8004eb0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.883 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ecc1d70b0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ec40019d0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 10 monclient(hunting): dump: 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:39.884 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3711358082 0 0) 0x7f5ecc1d70b0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 >> v1:172.21.15.145:6790/0 conn(0x7f5ecc0af900 legacy=0x7f5ecc1398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:39.885 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.878+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 >> v1:172.21.15.203:6789/0 conn(0x7f5ecc13ee90 legacy=0x7f5ecc06e9c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:39.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5eb3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:45:39.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5eb3fff640 1 monclient: found mon.a 2024-03-29T11:45:39.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5eb3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:39.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5eb3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:39.886 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ecc13a000 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5eb3fff640 10 monclient: _finish_auth 0 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5eb3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5eb3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5ed1d54640 5 monclient: authenticate success, global_id 4614 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5ed1d54640 10 monclient: _renew_subs 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5ed1d54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc13a250 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5ed1d54640 10 client.4614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5ed1d54640 10 monclient: _renew_subs 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5ed1d54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc142370 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.879+0000 7f5ed1d54640 1 librados: init done 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.880+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ec4004e10 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.887 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.880+0000 7f5eb3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.880+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5ec40051e0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.881+0000 7f5eb3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.881+0000 7f5eb3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.881+0000 7f5eb3fff640 10 monclient: dump: 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:39.888 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.881+0000 7f5ed1d54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:39.889 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.881+0000 7f5ec9ffb640 4 set_mon_vals no callback set 2024-03-29T11:45:39.889 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.882+0000 7f5ed1d54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:39.889 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.882+0000 7f5ed1d54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:39.889 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.882+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc075f10 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.889 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.883+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f5ec40053c0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.883+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 809964202 0 0) 0x7f5ec4084fd0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.883+0000 7f5eb3fff640 10 client.4614.objecter ms_dispatch 0x7f5ecc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:39.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.883+0000 7f5eb3fff640 3 client.4614.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:39.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.883+0000 7f5eb3fff640 3 client.4614.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:39.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.883+0000 7f5eb3fff640 20 client.4614.objecter dump_active .. 0 homeless 2024-03-29T11:45:39.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.884+0000 7f5ecaffd640 10 monclient: get_auth_request con 0x7f5eb806dbb0 auth_method 0 2024-03-29T11:45:39.890 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.884+0000 7f5eb3fff640 10 client.4614.objecter ms_handle_connect 0x7f5eb806dbb0 2024-03-29T11:45:39.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.894+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ec40508b0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:39.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.894+0000 7f5eb3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:39.898 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:39.894+0000 7f5eb3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:40.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.219+0000 7f5ed1d54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:40.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.219+0000 7f5ed1d54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:40.224 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.219+0000 7f5ed1d54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:40.224 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.219+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ecc0abcf0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:40.224 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.220+0000 7f5eb3fff640 1 -- 172.21.15.145:0/3009777914 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3238828718) 0x7f5ec40527e0 con 0x7f5ecc0a9ac0 2024-03-29T11:45:40.224 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.220+0000 7f5eb3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:40.225 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.220+0000 7f5eb3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:40.225 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:45:40.225 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":11,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:45:09.876003+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:09.876003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 10 librados: watch_flush enter 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 10 librados: watch_flush exit 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 20 client.4614.objecter shutdown clearing up homeless session... 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 10 client.4614.objecter successfully canceled tick 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 >> v1:172.21.15.145:6816/227372239 conn(0x7f5eb806dbb0 legacy=0x7f5eb8070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 10 monclient: shutdown 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 >> v1:172.21.15.145:6789/0 conn(0x7f5ecc0a9ac0 legacy=0x7f5ecc1d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 shutdown_connections 2024-03-29T11:45:40.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.224+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 >> 172.21.15.145:0/3009777914 conn(0x7f5ecc0b35e0 msgr2=0x7f5ecc13b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:40.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.225+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 shutdown_connections 2024-03-29T11:45:40.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.225+0000 7f5ed1d54640 1 -- 172.21.15.145:0/3009777914 wait complete. 2024-03-29T11:45:40.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:40.225+0000 7f5ed1d54640 1 librados: shutdown 2024-03-29T11:45:40.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:47.054 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:47.201 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.197+0000 7f0d38cf8640 10 monclient: get_monmap_and_config 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.197+0000 7f0d38cf8640 10 monclient: build_initial_monmap 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.197+0000 7f0d38cf8640 10 monclient: monmap: 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:45:47.198047+0000 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:45:47.198047+0000 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:45:47.202 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:47.203 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.199+0000 7f0d38cf8640 1 Processor -- start 2024-03-29T11:45:47.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.199+0000 7f0d38cf8640 1 -- start start 2024-03-29T11:45:47.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.199+0000 7f0d38cf8640 10 monclient: init 2024-03-29T11:45:47.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.200+0000 7f0d38cf8640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:47.204 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.200+0000 7f0d38cf8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:45:47.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.201+0000 7f0d38cf8640 10 monclient(hunting): picked mon.noname-b con 0x7f0d340b46b0 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:47.207 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.204+0000 7f0d38cf8640 10 monclient(hunting): picked mon.noname-c con 0x7f0d340aeae0 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:47.208 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.204+0000 7f0d32d9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f0d3413fa10 0x7f0d3414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:47652/0 (socket says 172.21.15.145:47652) 2024-03-29T11:45:47.209 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.204+0000 7f0d32d9d640 1 -- 172.21.15.145:0/3932882038 learned_addr learned my addr 172.21.15.145:0/3932882038 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:47.209 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.204+0000 7f0d38cf8640 10 monclient(hunting): picked mon.noname-a con 0x7f0d3413fa10 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:47.209 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.204+0000 7f0d38cf8640 1 -- 172.21.15.145:0/3932882038 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340b06a0 con 0x7f0d3413fa10 2024-03-29T11:45:47.209 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.204+0000 7f0d38cf8640 1 -- 172.21.15.145:0/3932882038 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340b4230 con 0x7f0d340aeae0 2024-03-29T11:45:47.209 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d38cf8640 1 -- 172.21.15.145:0/3932882038 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340b0910 con 0x7f0d340b46b0 2024-03-29T11:45:47.209 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d38cf8640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:47.209 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d38cf8640 10 monclient(hunting): authenticate will time out at 1536.512123s 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307660785 0 0) 0x7f0d340b06a0 con 0x7f0d3413fa10 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 10 monclient(hunting): my global_id is 4617 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d1c003520 con 0x7f0d3413fa10 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686474785 0 0) 0x7f0d340b4230 con 0x7f0d340aeae0 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 10 monclient(hunting): _negotiate 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:47.214 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 10 monclient(hunting): my global_id is 4603 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d340b06a0 con 0x7f0d340aeae0 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1559874793 0 0) 0x7f0d1c003520 con 0x7f0d3413fa10 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 >> v1:172.21.15.145:6790/0 conn(0x7f0d340b46b0 legacy=0x7f0d340b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.205+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 >> v1:172.21.15.203:6789/0 conn(0x7f0d340aeae0 legacy=0x7f0d3413f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: _finish_hunting 0 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 1 monclient: found mon.noname-a 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d340af6d0 con 0x7f0d3413fa10 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: _finish_auth 0 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: _check_auth_tickets 2024-03-29T11:45:47.215 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0d2c001a10 con 0x7f0d3413fa10 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: dump: 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d2c001c30 con 0x7f0d3413fa10 2024-03-29T11:45:47.216 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 1 -- 172.21.15.145:0/3932882038 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0d2c001fc0 con 0x7f0d3413fa10 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.206+0000 7f0d3159a640 10 monclient: dump: 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:47.217 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.207+0000 7f0d38cf8640 5 monclient: authenticate success, global_id 4617 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.207+0000 7f0d38cf8640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.207+0000 7f0d38cf8640 4 set_mon_vals no callback set 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.207+0000 7f0d38cf8640 10 monclient: shutdown 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.207+0000 7f0d38cf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.207+0000 7f0d38cf8640 1 -- 172.21.15.145:0/3932882038 >> v1:172.21.15.145:6789/0 conn(0x7f0d3413fa10 legacy=0x7f0d3414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.208+0000 7f0d38cf8640 1 -- 172.21.15.145:0/3932882038 shutdown_connections 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.208+0000 7f0d38cf8640 1 -- 172.21.15.145:0/3932882038 >> 172.21.15.145:0/3932882038 conn(0x7f0d34066a00 msgr2=0x7f0d34066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.208+0000 7f0d38cf8640 1 -- 172.21.15.145:0/3932882038 shutdown_connections 2024-03-29T11:45:47.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.209+0000 7f0d38cf8640 1 -- 172.21.15.145:0/3932882038 wait complete. 2024-03-29T11:45:47.226 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.210+0000 7f0d38cf8640 10 monclient: build_initial_monmap 2024-03-29T11:45:47.226 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.210+0000 7f0d38cf8640 10 monclient: monmap: 2024-03-29T11:45:47.226 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:47.226 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:47.226 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.211+0000 7f0d38cf8640 1 librados: starting msgr at 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.211+0000 7f0d38cf8640 1 librados: starting objecter 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.211+0000 7f0d38cf8640 1 Processor -- start 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.211+0000 7f0d38cf8640 1 -- start start 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.211+0000 7f0d38cf8640 1 librados: setting wanted keys 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.211+0000 7f0d38cf8640 1 librados: calling monclient init 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.211+0000 7f0d38cf8640 10 monclient: init 2024-03-29T11:45:47.227 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.213+0000 7f0d38cf8640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.213+0000 7f0d38cf8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.213+0000 7f0d38cf8640 10 monclient(hunting): picked mon.noname-c con 0x7f0d340aeae0 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.214+0000 7f0d38cf8640 10 monclient(hunting): picked mon.noname-a con 0x7f0d340b46b0 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.214+0000 7f0d31d9b640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f0d340b46b0 0x7f0d34142710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:49106/0 (socket says 172.21.15.145:49106) 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.214+0000 7f0d31d9b640 1 -- 172.21.15.145:0/2601102957 learned_addr learned my addr 172.21.15.145:0/2601102957 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.214+0000 7f0d38cf8640 10 monclient(hunting): picked mon.noname-b con 0x7f0d3413fa10 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.214+0000 7f0d38cf8640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d341447c0 con 0x7f0d340b46b0 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.214+0000 7f0d38cf8640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d34145380 con 0x7f0d3413fa10 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.214+0000 7f0d38cf8640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d341434d0 con 0x7f0d340aeae0 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.214+0000 7f0d38cf8640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.214+0000 7f0d38cf8640 10 monclient(hunting): authenticate will time out at 1536.521732s 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 client.?.objecter ms_handle_connect 0x7f0d340b46b0 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181811083 0 0) 0x7f0d341447c0 con 0x7f0d340b46b0 2024-03-29T11:45:47.228 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 monclient(hunting): my global_id is 4620 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14004eb0 con 0x7f0d340b46b0 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1067691173 0 0) 0x7f0d14004eb0 con 0x7f0d340b46b0 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d341447c0 con 0x7f0d340b46b0 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0d280019f0 con 0x7f0d340b46b0 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 monclient(hunting): dump: 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:47.229 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.215+0000 7f0d22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.216+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3512927910 0 0) 0x7f0d341447c0 con 0x7f0d340b46b0 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.216+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 >> v1:172.21.15.145:6790/0 conn(0x7f0d340aeae0 legacy=0x7f0d34144c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.216+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 >> v1:172.21.15.203:6789/0 conn(0x7f0d3413fa10 legacy=0x7f0d341deb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d22ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d22ffd640 1 monclient: found mon.a 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d22ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34143660 con 0x7f0d340b46b0 2024-03-29T11:45:47.230 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d22ffd640 10 monclient: _finish_auth 0 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d22ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d38cf8640 5 monclient: authenticate success, global_id 4620 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d38cf8640 10 monclient: _renew_subs 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d38cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.217+0000 7f0d38cf8640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3413b250 con 0x7f0d340b46b0 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d28001d10 con 0x7f0d340b46b0 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f0d28003a60 con 0x7f0d340b46b0 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d22ffd640 10 monclient: dump: 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:47.231 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d38cf8640 10 client.4620.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d38cf8640 10 monclient: _renew_subs 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d38cf8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d38cf8640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d341d6820 con 0x7f0d340b46b0 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.218+0000 7f0d38cf8640 1 librados: init done 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.220+0000 7f0d23fff640 4 set_mon_vals no callback set 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.221+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f0d28003c40 con 0x7f0d340b46b0 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.221+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 809964202 0 0) 0x7f0d280549e0 con 0x7f0d340b46b0 2024-03-29T11:45:47.232 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.221+0000 7f0d22ffd640 10 client.4620.objecter ms_dispatch 0x7f0d34000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:47.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.221+0000 7f0d22ffd640 3 client.4620.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:47.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.221+0000 7f0d22ffd640 3 client.4620.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:47.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.221+0000 7f0d22ffd640 20 client.4620.objecter dump_active .. 0 homeless 2024-03-29T11:45:47.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.222+0000 7f0d20ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:47.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.222+0000 7f0d20ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:47.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.222+0000 7f0d20ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:47.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.222+0000 7f0d20ff9640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d18005080 con 0x7f0d340b46b0 2024-03-29T11:45:47.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.222+0000 7f0d3259c640 10 monclient: get_auth_request con 0x7f0d1406dc00 auth_method 0 2024-03-29T11:45:47.233 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.222+0000 7f0d22ffd640 10 client.4620.objecter ms_handle_connect 0x7f0d1406dc00 2024-03-29T11:45:47.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.231+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d280504f0 con 0x7f0d340b46b0 2024-03-29T11:45:47.234 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.231+0000 7f0d22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:47.235 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.231+0000 7f0d22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:47.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.555+0000 7f0d20ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:47.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.555+0000 7f0d20ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:47.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.555+0000 7f0d20ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:45:47.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.555+0000 7f0d20ff9640 1 -- 172.21.15.145:0/2601102957 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d18005330 con 0x7f0d340b46b0 2024-03-29T11:45:47.560 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.556+0000 7f0d22ffd640 1 -- 172.21.15.145:0/2601102957 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3238828718) 0x7f0d28052420 con 0x7f0d340b46b0 2024-03-29T11:45:47.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.556+0000 7f0d22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:47.561 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.556+0000 7f0d22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:47.561 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:45:47.561 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":11,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:45:09.876003+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:09.876003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:47.564 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.559+0000 7f0d20ff9640 10 librados: watch_flush enter 2024-03-29T11:45:47.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 10 librados: watch_flush exit 2024-03-29T11:45:47.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 20 client.4620.objecter shutdown clearing up homeless session... 2024-03-29T11:45:47.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 10 client.4620.objecter successfully canceled tick 2024-03-29T11:45:47.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 1 -- 172.21.15.145:0/2601102957 >> v1:172.21.15.145:6816/227372239 conn(0x7f0d1406dc00 legacy=0x7f0d140700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:47.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 10 monclient: shutdown 2024-03-29T11:45:47.565 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:47.566 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 1 -- 172.21.15.145:0/2601102957 >> v1:172.21.15.145:6789/0 conn(0x7f0d340b46b0 legacy=0x7f0d34142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:47.566 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 1 -- 172.21.15.145:0/2601102957 shutdown_connections 2024-03-29T11:45:47.566 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 1 -- 172.21.15.145:0/2601102957 >> 172.21.15.145:0/2601102957 conn(0x7f0d34066a00 msgr2=0x7f0d3413f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:47.566 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 1 -- 172.21.15.145:0/2601102957 shutdown_connections 2024-03-29T11:45:47.566 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.560+0000 7f0d20ff9640 1 -- 172.21.15.145:0/2601102957 wait complete. 2024-03-29T11:45:47.566 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:47.561+0000 7f0d20ff9640 1 librados: shutdown 2024-03-29T11:45:47.582 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:45:54.387 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:45:54.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.514+0000 7f8252a7b640 10 monclient: get_monmap_and_config 2024-03-29T11:45:54.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.514+0000 7f8252a7b640 10 monclient: build_initial_monmap 2024-03-29T11:45:54.519 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.514+0000 7f8252a7b640 10 monclient: monmap: 2024-03-29T11:45:54.519 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:54.519 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:54.519 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:45:54.515518+0000 2024-03-29T11:45:54.520 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:45:54.515518+0000 2024-03-29T11:45:54.520 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:54.520 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:54.520 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:54.520 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:45:54.520 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:45:54.520 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:54.520 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.515+0000 7f8252a7b640 1 Processor -- start 2024-03-29T11:45:54.520 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.515+0000 7f8252a7b640 1 -- start start 2024-03-29T11:45:54.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.515+0000 7f8252a7b640 10 monclient: init 2024-03-29T11:45:54.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.516+0000 7f8252a7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:54.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.516+0000 7f8252a7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:45:54.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.516+0000 7f8252a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f824c1457f0 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:54.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.516+0000 7f8252a7b640 10 monclient(hunting): picked mon.noname-b con 0x7f824c13f650 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:54.521 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.517+0000 7f8252a7b640 10 monclient(hunting): picked mon.noname-c con 0x7f824c0a9d70 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:54.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.517+0000 7f8252a7b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c0aaef0 con 0x7f824c1457f0 2024-03-29T11:45:54.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.517+0000 7f8252a7b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c145bd0 con 0x7f824c0a9d70 2024-03-29T11:45:54.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.517+0000 7f8252a7b640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c145eb0 con 0x7f824c13f650 2024-03-29T11:45:54.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.517+0000 7f8252a7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:54.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.517+0000 7f8252a7b640 10 monclient(hunting): authenticate will time out at 1543.825195s 2024-03-29T11:45:54.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.517+0000 7f8251018640 1 --1- >> v1:172.21.15.203:6789/0 conn(0x7f824c0a9d70 0x7f824c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.203:6789/0 says I am v1:172.21.15.145:60748/0 (socket says 172.21.15.145:60748) 2024-03-29T11:45:54.522 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.517+0000 7f8251018640 1 -- 172.21.15.145:0/1504982828 learned_addr learned my addr 172.21.15.145:0/1504982828 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:54.523 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 801155119 0 0) 0x7f824c145eb0 con 0x7f824c13f650 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8234003520 con 0x7f824c13f650 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654224959 0 0) 0x7f824c145bd0 con 0x7f824c0a9d70 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 10 monclient(hunting): my global_id is 4606 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.518+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f824c145eb0 con 0x7f824c0a9d70 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.519+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1963077079 0 0) 0x7f8234003520 con 0x7f824c13f650 2024-03-29T11:45:54.524 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.519+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 >> v1:172.21.15.203:6789/0 conn(0x7f824c0a9d70 legacy=0x7f824c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:54.525 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.519+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 >> v1:172.21.15.145:6789/0 conn(0x7f824c1457f0 legacy=0x7f824c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.145:6790/0 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c146340 con 0x7f824c13f650 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f8252a7b640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:45:54.526 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f8252a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8240001940 con 0x7f824c13f650 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f824b7fe640 10 monclient: dump: 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:54.527 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.520+0000 7f8252a7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:45:54.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.521+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 <== mon.2 v1:172.21.15.145:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8240001aa0 con 0x7f824c13f650 2024-03-29T11:45:54.528 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.521+0000 7f824b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:54.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.521+0000 7f824b7fe640 1 -- 172.21.15.145:0/1504982828 <== mon.2 v1:172.21.15.145:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f8240001eb0 con 0x7f824c13f650 2024-03-29T11:45:54.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.521+0000 7f824b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:54.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.521+0000 7f824b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:54.530 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.521+0000 7f824b7fe640 10 monclient: dump: 2024-03-29T11:45:54.530 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:54.530 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:54.530 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.521+0000 7f8252a7b640 10 monclient: get_monmap_and_config success 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.521+0000 7f8252a7b640 4 set_mon_vals no callback set 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.522+0000 7f8252a7b640 10 monclient: shutdown 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.522+0000 7f8252a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.522+0000 7f8252a7b640 1 -- 172.21.15.145:0/1504982828 >> v1:172.21.15.145:6790/0 conn(0x7f824c13f650 legacy=0x7f824c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.522+0000 7f8252a7b640 1 -- 172.21.15.145:0/1504982828 shutdown_connections 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.522+0000 7f8252a7b640 1 -- 172.21.15.145:0/1504982828 >> 172.21.15.145:0/1504982828 conn(0x7f824c066a00 msgr2=0x7f824c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:54.531 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.522+0000 7f8252a7b640 1 -- 172.21.15.145:0/1504982828 shutdown_connections 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.522+0000 7f8252a7b640 1 -- 172.21.15.145:0/1504982828 wait complete. 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.523+0000 7f8252a7b640 10 monclient: build_initial_monmap 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.523+0000 7f8252a7b640 10 monclient: monmap: 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.523+0000 7f8252a7b640 1 librados: starting msgr at 2024-03-29T11:45:54.532 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.523+0000 7f8252a7b640 1 librados: starting objecter 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.523+0000 7f8252a7b640 1 Processor -- start 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.523+0000 7f8252a7b640 1 -- start start 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.523+0000 7f8252a7b640 1 librados: setting wanted keys 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.523+0000 7f8252a7b640 1 librados: calling monclient init 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.523+0000 7f8252a7b640 10 monclient: init 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.524+0000 7f8252a7b640 10 monclient: _reopen_session rank -1 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.524+0000 7f8252a7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.524+0000 7f8252a7b640 10 monclient(hunting): picked mon.noname-a con 0x7f824c0a9d70 addr v1:172.21.15.145:6789/0 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.524+0000 7f8252a7b640 10 monclient(hunting): picked mon.noname-b con 0x7f824c13f650 addr v1:172.21.15.203:6789/0 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.524+0000 7f8252a7b640 10 monclient(hunting): picked mon.noname-c con 0x7f824c1457f0 addr v1:172.21.15.145:6790/0 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.524+0000 7f8252a7b640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c1db190 con 0x7f824c0a9d70 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.525+0000 7f8252a7b640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c141ff0 con 0x7f824c13f650 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.525+0000 7f8252a7b640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f824c1404b0 con 0x7f824c1457f0 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.525+0000 7f8252a7b640 10 monclient(hunting): _renew_subs 2024-03-29T11:45:54.533 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.525+0000 7f8252a7b640 10 monclient(hunting): authenticate will time out at 1543.833092s 2024-03-29T11:45:54.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.525+0000 7f8251018640 1 --1- >> v1:172.21.15.145:6790/0 conn(0x7f824c1457f0 0x7f824c1e2e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6790/0 says I am v1:172.21.15.145:53790/0 (socket says 172.21.15.145:53790) 2024-03-29T11:45:54.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.525+0000 7f8251018640 1 -- 172.21.15.145:0/2968072178 learned_addr learned my addr 172.21.15.145:0/2968072178 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:45:54.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.525+0000 7f82497fa640 10 client.?.objecter ms_handle_connect 0x7f824c1457f0 2024-03-29T11:45:54.534 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.525+0000 7f82497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:54.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.525+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721249682 0 0) 0x7f824c1404b0 con 0x7f824c1457f0 2024-03-29T11:45:54.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:54.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:54.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:54.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:45:54.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8238004eb0 con 0x7f824c1457f0 2024-03-29T11:45:54.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2200802287 0 0) 0x7f8238004eb0 con 0x7f824c1457f0 2024-03-29T11:45:54.539 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f824c1404b0 con 0x7f824c1457f0 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 client.?.objecter ms_handle_connect 0x7f824c13f650 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f823c001930 con 0x7f824c1457f0 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 monclient(hunting): dump: 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:54.540 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.526+0000 7f82497fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.527+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3210622027 0 0) 0x7f824c141ff0 con 0x7f824c13f650 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.527+0000 7f82497fa640 10 monclient(hunting): _negotiate 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.527+0000 7f82497fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.527+0000 7f82497fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.527+0000 7f82497fa640 10 monclient(hunting): my global_id is 4609 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.527+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8238004eb0 con 0x7f824c13f650 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.527+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 209276945 0 0) 0x7f824c1404b0 con 0x7f824c1457f0 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.527+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 >> v1:172.21.15.203:6789/0 conn(0x7f824c13f650 legacy=0x7f824c0b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.527+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 >> v1:172.21.15.145:6789/0 conn(0x7f824c0a9d70 legacy=0x7f824c13fda0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 10 monclient: _finish_hunting 0 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 1 monclient: found mon.c 2024-03-29T11:45:54.541 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 --> v1:172.21.15.145:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f824c142260 con 0x7f824c1457f0 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 10 monclient: _finish_auth 0 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 10 monclient: _check_auth_tickets 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f8252a7b640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f8252a7b640 10 monclient: _renew_subs 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f8252a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 --> v1:172.21.15.145:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824c13bac0 con 0x7f824c1457f0 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f823c001b90 con 0x7f824c1457f0 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f82497fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.528+0000 7f824affd640 4 set_mon_vals no callback set 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.529+0000 7f8252a7b640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:45:54.542 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.529+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f823c004c30 con 0x7f824c1457f0 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.529+0000 7f8252a7b640 10 monclient: _renew_subs 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.529+0000 7f8252a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.529+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 --> v1:172.21.15.145:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f824c0b0f40 con 0x7f824c1457f0 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.529+0000 7f8252a7b640 1 librados: init done 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.529+0000 7f82497fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.530+0000 7f82497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.530+0000 7f82497fa640 10 monclient: dump: 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:45:54.543 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.530+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f823c01ac40 con 0x7f824c1457f0 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.531+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 809964202 0 0) 0x7f823c084c50 con 0x7f824c1457f0 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.531+0000 7f82497fa640 10 client.4643.objecter ms_dispatch 0x7f824c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.531+0000 7f82497fa640 3 client.4643.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.531+0000 7f82497fa640 3 client.4643.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.531+0000 7f82497fa640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.531+0000 7f8250817640 10 monclient: get_auth_request con 0x7f823806dc60 auth_method 0 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.532+0000 7f82497fa640 10 client.4643.objecter ms_handle_connect 0x7f823806dc60 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.532+0000 7f8252a7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.532+0000 7f8252a7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.532+0000 7f8252a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:54.544 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.532+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824c135ad0 con 0x7f824c1457f0 2024-03-29T11:45:54.545 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.541+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f823c0503c0 con 0x7f824c1457f0 2024-03-29T11:45:54.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.541+0000 7f82497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:45:54.546 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.541+0000 7f82497fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:45:54.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.846+0000 7f8252a7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:54.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.846+0000 7f8252a7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:54.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.846+0000 7f8252a7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.145:6790/0 2024-03-29T11:45:54.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.846+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 --> v1:172.21.15.145:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f824c075f10 con 0x7f824c1457f0 2024-03-29T11:45:54.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.847+0000 7f82497fa640 1 -- 172.21.15.145:0/2968072178 <== mon.2 v1:172.21.15.145:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3238828718) 0x7f823c084300 con 0x7f824c1457f0 2024-03-29T11:45:54.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.847+0000 7f82497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:45:54.853 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.847+0000 7f82497fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:45:54.853 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:45:54.853 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":11,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:45:09.876003+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:09.876003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:45:54.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 10 librados: watch_flush enter 2024-03-29T11:45:54.855 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 10 librados: watch_flush exit 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 >> v1:172.21.15.145:6816/227372239 conn(0x7f823806dc60 legacy=0x7f8238070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 10 monclient: shutdown 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 >> v1:172.21.15.145:6790/0 conn(0x7f824c1457f0 legacy=0x7f824c1e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 shutdown_connections 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.850+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 >> 172.21.15.145:0/2968072178 conn(0x7f824c066a00 msgr2=0x7f824c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.851+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 shutdown_connections 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.851+0000 7f8252a7b640 1 -- 172.21.15.145:0/2968072178 wait complete. 2024-03-29T11:45:54.856 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:45:54.851+0000 7f8252a7b640 1 librados: shutdown 2024-03-29T11:45:54.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:01.677 DEBUG:teuthology.orchestra.run.smithi145:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:46:01.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.818+0000 7f5a9359e640 10 monclient: get_monmap_and_config 2024-03-29T11:46:01.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.818+0000 7f5a9359e640 10 monclient: build_initial_monmap 2024-03-29T11:46:01.824 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.818+0000 7f5a9359e640 10 monclient: monmap: 2024-03-29T11:46:01.824 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:46:01.824 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:46:01.824 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:46:01.820158+0000 2024-03-29T11:46:01.824 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:46:01.820158+0000 2024-03-29T11:46:01.824 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:01.824 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:46:01.825 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:46:01.825 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-c 2024-03-29T11:46:01.825 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-b 2024-03-29T11:46:01.825 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:46:01.825 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.819+0000 7f5a9359e640 1 Processor -- start 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.820+0000 7f5a9359e640 1 -- start start 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.820+0000 7f5a9359e640 10 monclient: init 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 10 monclient(hunting): picked mon.noname-a con 0x7f5a8c02b570 addr v1:172.21.15.145:6789/0 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 10 monclient(hunting): picked mon.noname-b con 0x7f5a8c006800 addr v1:172.21.15.145:6790/0 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8c006c80 addr v1:172.21.15.203:6789/0 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c032ef0 con 0x7f5a8c02b570 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 1 -- --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c0be170 con 0x7f5a8c006c80 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 1 -- --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c02c720 con 0x7f5a8c006800 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9359e640 10 monclient(hunting): authenticate will time out at 1551.129277s 2024-03-29T11:46:01.826 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9259c640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5a8c02b570 0x7f5a8c0345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:34284/0 (socket says 172.21.15.145:34284) 2024-03-29T11:46:01.827 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.821+0000 7f5a9259c640 1 -- 172.21.15.145:0/3784442709 learned_addr learned my addr 172.21.15.145:0/3784442709 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:46:01.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806494847 0 0) 0x7f5a88001110 con 0x7f5a8c02b570 2024-03-29T11:46:01.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient(hunting): my global_id is 4623 2024-03-29T11:46:01.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a7c003520 con 0x7f5a8c02b570 2024-03-29T11:46:01.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683499155 0 0) 0x7f5a8c02c720 con 0x7f5a8c006800 2024-03-29T11:46:01.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.830 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a88001110 con 0x7f5a8c006800 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1104505330 0 0) 0x7f5a7c003520 con 0x7f5a8c02b570 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 >> v1:172.21.15.145:6790/0 conn(0x7f5a8c006800 legacy=0x7f5a8c0bda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 >> v1:172.21.15.203:6789/0 conn(0x7f5a8c006c80 legacy=0x7f5a8c0c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient: _finish_hunting 0 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 monclient: found mon.noname-a 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.145:6789/0 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c02c990 con 0x7f5a8c02b570 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient: _finish_auth 0 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient: _check_auth_tickets 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:01.831 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a88001680 con 0x7f5a8c02b570 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9159a640 10 monclient: dump: 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9359e640 5 monclient: authenticate success, global_id 4623 2024-03-29T11:46:01.832 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.822+0000 7f5a9359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.823+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 <== mon.0 v1:172.21.15.145:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a880017e0 con 0x7f5a8c02b570 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.823+0000 7f5a9159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.823+0000 7f5a9159a640 1 -- 172.21.15.145:0/3784442709 <== mon.0 v1:172.21.15.145:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a88001bf0 con 0x7f5a8c02b570 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.823+0000 7f5a9159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.823+0000 7f5a9159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.823+0000 7f5a9159a640 10 monclient: dump: 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:46:01.833 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.823+0000 7f5a9359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.823+0000 7f5a9359e640 4 set_mon_vals no callback set 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.824+0000 7f5a9359e640 10 monclient: shutdown 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.824+0000 7f5a9359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.824+0000 7f5a9359e640 1 -- 172.21.15.145:0/3784442709 >> v1:172.21.15.145:6789/0 conn(0x7f5a8c02b570 legacy=0x7f5a8c0345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.824+0000 7f5a9359e640 1 -- 172.21.15.145:0/3784442709 shutdown_connections 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.824+0000 7f5a9359e640 1 -- 172.21.15.145:0/3784442709 >> 172.21.15.145:0/3784442709 conn(0x7f5a8c0058f0 msgr2=0x7f5a8c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.824+0000 7f5a9359e640 1 -- 172.21.15.145:0/3784442709 shutdown_connections 2024-03-29T11:46:01.834 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.824+0000 7f5a9359e640 1 -- 172.21.15.145:0/3784442709 wait complete. 2024-03-29T11:46:01.841 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.825+0000 7f5a9359e640 10 monclient: build_initial_monmap 2024-03-29T11:46:01.841 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.825+0000 7f5a9359e640 10 monclient: monmap: 2024-03-29T11:46:01.841 INFO:teuthology.misc.health.smithi145.stderr:epoch 0 2024-03-29T11:46:01.841 INFO:teuthology.misc.health.smithi145.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:46:01.841 INFO:teuthology.misc.health.smithi145.stderr:last_changed 0.000000 2024-03-29T11:46:01.841 INFO:teuthology.misc.health.smithi145.stderr:created 0.000000 2024-03-29T11:46:01.841 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 0 (unknown) 2024-03-29T11:46:01.841 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:46:01.841 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.noname-a 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.noname-b 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.noname-c 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.826+0000 7f5a9359e640 1 librados: starting msgr at 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.826+0000 7f5a9359e640 1 librados: starting objecter 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.826+0000 7f5a9359e640 1 Processor -- start 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.826+0000 7f5a9359e640 1 -- start start 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.826+0000 7f5a9359e640 1 librados: setting wanted keys 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.826+0000 7f5a9359e640 1 librados: calling monclient init 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.826+0000 7f5a9359e640 10 monclient: init 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.827+0000 7f5a9359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.827+0000 7f5a9359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.827+0000 7f5a9359e640 10 monclient(hunting): picked mon.noname-b con 0x7f5a8c006800 addr v1:172.21.15.203:6789/0 2024-03-29T11:46:01.842 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.827+0000 7f5a9359e640 10 monclient(hunting): picked mon.noname-c con 0x7f5a8c006c80 addr v1:172.21.15.145:6790/0 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.827+0000 7f5a9359e640 10 monclient(hunting): picked mon.noname-a con 0x7f5a8c02b570 addr v1:172.21.15.145:6789/0 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.827+0000 7f5a9359e640 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c0c4da0 con 0x7f5a8c02b570 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.827+0000 7f5a92d9d640 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5a8c02b570 0x7f5a8c0c34e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:34570/0 (socket says 172.21.15.145:34570) 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.827+0000 7f5a92d9d640 1 -- 172.21.15.145:0/372270846 learned_addr learned my addr 172.21.15.145:0/372270846 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.827+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.203:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c02d210 con 0x7f5a8c006800 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 client.?.objecter ms_handle_connect 0x7f5a8c02b570 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 client.?.objecter ms_handle_connect 0x7f5a8c006c80 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a8c02d4f0 con 0x7f5a8c006c80 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a9359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a9359e640 10 monclient(hunting): authenticate will time out at 1551.136424s 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 client.?.objecter ms_handle_connect 0x7f5a8c006800 2024-03-29T11:46:01.843 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628954594 0 0) 0x7f5a8c0c4da0 con 0x7f5a8c02b570 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 monclient(hunting): my global_id is 4626 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a74004eb0 con 0x7f5a8c02b570 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.2 v1:172.21.15.145:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384696783 0 0) 0x7f5a8c02d4f0 con 0x7f5a8c006c80 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.828+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a8c0c4da0 con 0x7f5a8c006c80 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.1 v1:172.21.15.203:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812174898 0 0) 0x7f5a8c02d210 con 0x7f5a8c006800 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:46:01.844 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 10 monclient(hunting): my global_id is 4612 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.203:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a8c02d4f0 con 0x7f5a8c006800 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2185635989 0 0) 0x7f5a74004eb0 con 0x7f5a8c02b570 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a8c02d210 con 0x7f5a8c02b570 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a9404fdd0 con 0x7f5a8c02b570 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 10 monclient(hunting): dump: 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:46:01.845 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.829+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.2 v1:172.21.15.145:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2864591244 0 0) 0x7f5a8c0c4da0 con 0x7f5a8c006c80 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a74004eb0 con 0x7f5a8c006c80 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.2 v1:172.21.15.145:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a840018b0 con 0x7f5a8c006c80 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient(hunting): dump: 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:46:01.846 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.1 v1:172.21.15.203:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4179380521 0 0) 0x7f5a8c02d4f0 con 0x7f5a8c006800 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.203:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a8c0c4da0 con 0x7f5a8c006800 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.1 v1:172.21.15.203:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a880047e0 con 0x7f5a8c006800 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient(hunting): dump: 2024-03-29T11:46:01.847 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1122686760 0 0) 0x7f5a8c02d210 con 0x7f5a8c02b570 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 >> v1:172.21.15.145:6790/0 conn(0x7f5a8c006c80 legacy=0x7f5a8c0c0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 >> v1:172.21.15.203:6789/0 conn(0x7f5a8c006800 legacy=0x7f5a8c02d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 monclient: found mon.a 2024-03-29T11:46:01.848 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a8c0c2d00 con 0x7f5a8c02b570 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient: _finish_auth 0 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a9359e640 5 monclient: authenticate success, global_id 4626 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a9359e640 10 monclient: _renew_subs 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a9359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a8c0c3fb0 con 0x7f5a8c02b570 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a9359e640 10 client.4626.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a9359e640 10 monclient: _renew_subs 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a9359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a8c16cb20 con 0x7f5a8c02b570 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.830+0000 7f5a9359e640 1 librados: init done 2024-03-29T11:46:01.849 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.831+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a9404f400 con 0x7f5a8c02b570 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.831+0000 7f5a82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.831+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2840026854 0 0) 0x7f5a94075710 con 0x7f5a8c02b570 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.831+0000 7f5a82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.831+0000 7f5a82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.831+0000 7f5a90d99640 4 set_mon_vals no callback set 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.831+0000 7f5a82ffd640 10 monclient: dump: 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:epoch 1 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:fsid 524911ae-c3df-44f6-b399-af269c927521 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:last_changed 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:created 2024-03-29T11:34:26.758276+0000 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:min_mon_release 19 (squid) 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:election_strategy: 1 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:0: v1:172.21.15.145:6789/0 mon.a 2024-03-29T11:46:01.850 INFO:teuthology.misc.health.smithi145.stderr:1: v1:172.21.15.203:6789/0 mon.b 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2: v1:172.21.15.145:6790/0 mon.c 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr: 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.833+0000 7f5a9359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.833+0000 7f5a9359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.833+0000 7f5a9359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.833+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a78005080 con 0x7f5a8c02b570 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.833+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 3469986450 0 0) 0x7f5a94067bf0 con 0x7f5a8c02b570 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.834+0000 7f5a9259c640 10 monclient: get_auth_request con 0x7f5a7406dfc0 auth_method 0 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.834+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 809964202 0 0) 0x7f5a940ff550 con 0x7f5a8c02b570 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.834+0000 7f5a82ffd640 10 client.4626.objecter ms_dispatch 0x7f5a94000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.834+0000 7f5a82ffd640 3 client.4626.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.834+0000 7f5a82ffd640 3 client.4626.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.834+0000 7f5a82ffd640 20 client.4626.objecter dump_active .. 0 homeless 2024-03-29T11:46:01.851 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.834+0000 7f5a82ffd640 10 client.4626.objecter ms_handle_connect 0x7f5a7406dfc0 2024-03-29T11:46:01.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.844+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a940cae30 con 0x7f5a8c02b570 2024-03-29T11:46:01.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.844+0000 7f5a82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:46:01.852 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:01.844+0000 7f5a82ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:46:02.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.213+0000 7f5a9359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:02.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.213+0000 7f5a9359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:02.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.213+0000 7f5a9359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.145:6789/0 2024-03-29T11:46:02.218 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.213+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 --> v1:172.21.15.145:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a78005330 con 0x7f5a8c02b570 2024-03-29T11:46:02.219 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.214+0000 7f5a82ffd640 1 -- 172.21.15.145:0/372270846 <== mon.0 v1:172.21.15.145:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3238828718) 0x7f5a940ccd60 con 0x7f5a8c02b570 2024-03-29T11:46:02.219 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.214+0000 7f5a82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:46:02.219 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.214+0000 7f5a82ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:46:02.219 INFO:teuthology.misc.health.smithi145.stdout: 2024-03-29T11:46:02.219 INFO:teuthology.misc.health.smithi145.stdout:{"epoch":11,"fsid":"524911ae-c3df-44f6-b399-af269c927521","created":"2024-03-29T11:35:04.946181+0000","modified":"2024-03-29T11:45:09.876003+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:45:09.876003+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3dc3165d-1d81-40d7-913b-378677dd8310","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"35f88c45-b7c4-406d-b24a-9b3f432b80ad","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"f33f2af2-bb94-43d0-96d0-6b99d0a736fa","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"42e5db63-c492-433f-810c-2e58ea1368cf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"3ff5f6e2-6f34-43d2-b346-4750c70d20f6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"655ce144-bcb6-4dce-9e52-fd2a51349f52","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"72515dad-d50c-4e13-bfc2-6ea629fbde5a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8a0b7d49-f607-418c-9c4e-44e6f25cd916","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:46:02.222 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.217+0000 7f5a9359e640 10 librados: watch_flush enter 2024-03-29T11:46:02.222 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.217+0000 7f5a9359e640 10 librados: watch_flush exit 2024-03-29T11:46:02.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.217+0000 7f5a9359e640 20 client.4626.objecter shutdown clearing up homeless session... 2024-03-29T11:46:02.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.217+0000 7f5a9359e640 10 client.4626.objecter successfully canceled tick 2024-03-29T11:46:02.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.218+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 >> v1:172.21.15.145:6816/227372239 conn(0x7f5a7406dfc0 legacy=0x7f5a74070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:02.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.218+0000 7f5a9359e640 10 monclient: shutdown 2024-03-29T11:46:02.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.218+0000 7f5a9359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:46:02.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.218+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 >> v1:172.21.15.145:6789/0 conn(0x7f5a8c02b570 legacy=0x7f5a8c0c34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:46:02.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.218+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 shutdown_connections 2024-03-29T11:46:02.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.218+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 >> 172.21.15.145:0/372270846 conn(0x7f5a8c0058f0 msgr2=0x7f5a8c029920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:46:02.223 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.218+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 shutdown_connections 2024-03-29T11:46:02.224 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.218+0000 7f5a9359e640 1 -- 172.21.15.145:0/372270846 wait complete. 2024-03-29T11:46:02.224 INFO:teuthology.misc.health.smithi145.stderr:2024-03-29T11:46:02.218+0000 7f5a9359e640 1 librados: shutdown 2024-03-29T11:46:02.238 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:46:02.238 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:46:02.240 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:46:02.240 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:46:02.240 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.271 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:46:02.271 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:46:02.271 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.341 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:46:02.342 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:46:02.342 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.410 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:46:02.410 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:46:02.410 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.485 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:46:02.486 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:46:02.486 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.525 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:46:02.525 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:46:02.526 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.609 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:46:02.610 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:46:02.610 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.639 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:46:02.639 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:46:02.639 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.716 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:46:02.717 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:46:02.717 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:46:02.717 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.741 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:46:02.741 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:46:02.741 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.781 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:46:02.781 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:46:02.782 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:46:02.782 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.793 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:46:02.793 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:46:02.793 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.846 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:46:02.846 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:46:02.846 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:46:02.857 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:46:02.858 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:46:02.858 DEBUG:teuthology.orchestra.run.smithi145:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:46:02.918 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:46:02.918 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:46:06.439 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:46:06.439 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:46:06.526 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:46:06.527 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:46:06.585 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:46:06.585 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:46:06.643 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:46:06.643 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:46:10.074 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:46:10.074 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:46:10.151 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:46:10.151 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:46:10.209 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:46:10.209 DEBUG:teuthology.orchestra.run.smithi203:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:46:10.268 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:46:10.269 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/data/mon.a.tgz 2024-03-29T11:46:10.269 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-03-29T11:46:10.287 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.UOZ9CiA1s5 2024-03-29T11:46:10.288 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.UOZ9CiA1s5 2024-03-29T11:46:10.432 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.UOZ9CiA1s5 2024-03-29T11:46:10.519 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.UOZ9CiA1s5 is 212KB 2024-03-29T11:46:10.613 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.UOZ9CiA1s5 2024-03-29T11:46:10.627 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/data/mon.c.tgz 2024-03-29T11:46:10.628 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-03-29T11:46:10.682 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.VAey6JkgGm 2024-03-29T11:46:10.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.VAey6JkgGm 2024-03-29T11:46:10.782 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.VAey6JkgGm 2024-03-29T11:46:10.867 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.VAey6JkgGm is 212KB 2024-03-29T11:46:10.959 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.VAey6JkgGm 2024-03-29T11:46:10.974 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/data/mon.b.tgz 2024-03-29T11:46:10.974 DEBUG:teuthology.orchestra.run.smithi203:> mktemp 2024-03-29T11:46:10.991 INFO:teuthology.orchestra.run.smithi203.stdout:/tmp/tmp.IIUh3SJfCp 2024-03-29T11:46:10.992 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.IIUh3SJfCp 2024-03-29T11:46:11.112 DEBUG:teuthology.orchestra.run.smithi203:> sudo chmod 0666 /tmp/tmp.IIUh3SJfCp 2024-03-29T11:46:11.200 DEBUG:teuthology.orchestra.remote:smithi203:/tmp/tmp.IIUh3SJfCp is 212KB 2024-03-29T11:46:11.294 DEBUG:teuthology.orchestra.run.smithi203:> rm -fr /tmp/tmp.IIUh3SJfCp 2024-03-29T11:46:11.309 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:46:11.309 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:46:11.311 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:46:11.392 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:46:11.432 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:46:11.432 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:46:11.434 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/remote/smithi145/crash 2024-03-29T11:46:11.435 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:46:11.462 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/remote/smithi203/crash 2024-03-29T11:46:11.463 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:46:11.499 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:46:11.499 DEBUG:teuthology.orchestra.run.smithi145:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:46:11.507 DEBUG:teuthology.orchestra.run.smithi203:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:46:11.570 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40392.log 2024-03-29T11:46:11.570 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39767.log 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40392.log: /var/log/ceph/ceph-client.admin.39767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39060.log 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-mon.a.log: 85.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40392.log.gz 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41328.log 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39767.log.gz 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41289.log 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40197.log 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39060.log.gz 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41289.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41328.log.gz 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41406.log 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40197.log: gzip 86.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.41289.log.gz --verbose 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr: -- /var/log/ceph/ceph-client.admin.40665.log 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39294.log 2024-03-29T11:46:11.571 INFO:teuthology.orchestra.run.smithi145.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40197.log.gz 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41367.log 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41406.log.gz 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39294.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40665.log.gz 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39177.log 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38982.log 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39294.log.gz 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39645.log 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41367.log.gz 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38982.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39177.log.gz 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39845.log 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38943.log 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38982.log.gz 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39806.log 2024-03-29T11:46:11.572 INFO:teuthology.orchestra.run.smithi145.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39645.log.gz 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38943.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39845.log.gz 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38083.log 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38943.log.gz 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41445.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39806.log.gz 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40079.log 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40977.log 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40118.log 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38083.log.gz 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40079.log.gz 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40860.log 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40977.log: /var/log/ceph/ceph-client.admin.40118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39255.log 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38161.log 2024-03-29T11:46:11.573 INFO:teuthology.orchestra.run.smithi145.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40977.log.gz 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40118.log.gz 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41211.log 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40860.log.gz 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38748.log 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39255.log.gz 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41211.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38161.log.gz 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40938.log 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41211.log.gz 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38631.log 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38748.log.gz 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39099.log 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:46:11.574 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40548.log 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38631.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40938.log.gz 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41055.log 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40548.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38631.log.gz 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38122.log 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39099.log.gz 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38904.log 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40548.log.gz 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39489.log 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38122.log: /var/log/ceph/ceph-client.admin.38904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40001.log 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41055.log.gz 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38122.log.gz 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39567.log 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38904.log.gz 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38553.log 2024-03-29T11:46:11.575 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40001.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39489.log.gz 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39884.log 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38553.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40001.log.gz 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- 86.0% /var/log/ceph/ceph-client.admin.39333.log 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39567.log.gz 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39884.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38553.log.gz 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41172.log 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39528.log 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39884.log.gz 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39333.log.gz 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40782.log 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39528.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28497.log 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41172.log.gz 2024-03-29T11:46:11.576 INFO:teuthology.orchestra.run.smithi145.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39528.log.gz 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38592.log 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph.tmp-client.admin.28497.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40314.log 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28497.log.gz 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40782.log.gz 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38592.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39138.log 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40314.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.38592.log.gz 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40275.log 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38787.log 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39138.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40314.log.gz 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38475.log 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38787.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39138.log.gz 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40704.log 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40275.log.gz 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38475.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38787.log.gz 2024-03-29T11:46:11.577 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38865.log 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38200.log 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38475.log.gz 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40704.log: /var/log/ceph/ceph-client.admin.38865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39962.log 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38200.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40704.log.gz 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr: 85.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.38865.log.gz -5 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40626.log 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38044.log 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38200.log.gz 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40626.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38044.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39962.log.gz 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39216.log 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40626.log.gz 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38044.log.gz 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38005.log 2024-03-29T11:46:11.578 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38670.log 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39216.log: /var/log/ceph/ceph-client.admin.38005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40470.log 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39216.log.gz 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38670.log: gzip 86.6% -5 --verbose -- /var/log/ceph/ceph-client.admin.40236.log 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38005.log.gz 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39021.log 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.38670.log.gz 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38280.log 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40470.log.gz 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41016.log 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40236.log.gz 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38280.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39021.log.gz 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40040.log 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40431.log 2024-03-29T11:46:11.579 INFO:teuthology.orchestra.run.smithi145.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38280.log.gz 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41016.log: /var/log/ceph/ceph-client.admin.40040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38826.log 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41016.log.gz 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40743.log 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40040.log.gz 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38826.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40431.log.gz 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38319.log 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40743.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38826.log.gz 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34700.log 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40743.log.gz 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38319.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34633.log 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.34700.log: gzip -5 --verbose -- 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38319.log.gz 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr: /var/log/ceph/ceph-client.admin.40587.log 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.34633.log: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39411.log 2024-03-29T11:46:11.580 INFO:teuthology.orchestra.run.smithi145.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34700.log.gz 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40587.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34633.log.gz 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40899.log 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40157.log 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40587.log.gz 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40899.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39411.log.gz 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38709.log 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40899.log.gz 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41250.log 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40157.log: /var/log/ceph/ceph-client.admin.38709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38358.log 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38514.log 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38709.log.gz 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40157.log.gz 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38358.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41250.log.gz 2024-03-29T11:46:11.581 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39450.log 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41094.log 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38358.log.gz 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38514.log: /var/log/ceph/ceph-client.admin.39450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39923.log 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38514.log.gz 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39450.log.gz/var/log/ceph/ceph-client.admin.41094.log: 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40353.log 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr: 86.4%/var/log/ceph/ceph-osd.0.log: -- replaced with /var/log/ceph/ceph-client.admin.41094.log.gz 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34306.log 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40353.log: -- /var/log/ceph/ceph.log 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.34306.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40353.log.gz 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34306.log.gz 2024-03-29T11:46:11.582 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38239.log 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-osd.2.log: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39723.log 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39606.log 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38239.log.gz 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39723.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39723.log.gz 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39684.log 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39684.log: /var/log/ceph/ceph-client.admin.39606.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39606.log.gz 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39684.log.gz 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40509.log 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38436.log 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.39372.log: /var/log/ceph/ceph-client.admin.40509.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40509.log.gz 2024-03-29T11:46:11.583 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41133.log 2024-03-29T11:46:11.584 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40821.log 2024-03-29T11:46:11.584 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.41133.log: /var/log/ceph/ceph-client.admin.38436.log: 85.9% 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38436.log.gz 2024-03-29T11:46:11.584 INFO:teuthology.orchestra.run.smithi145.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41133.log.gz 2024-03-29T11:46:11.584 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:46:11.584 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38397.log 2024-03-29T11:46:11.584 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.40821.log: /var/log/ceph/ceph-mon.c.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40821.log.gz 2024-03-29T11:46:11.584 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/ceph-client.admin.38397.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38397.log.gz 2024-03-29T11:46:11.590 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:46:11.590 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34610.log 2024-03-29T11:46:11.590 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34810.log 2024-03-29T11:46:11.590 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34610.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:46:11.590 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34810.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:46:11.590 INFO:teuthology.orchestra.run.smithi203.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34610.log.gz 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34770.log 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34810.log.gz 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.4.log: 88.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34530.log 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34770.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34530.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34770.log.gz 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34530.log.gz 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34650.log 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34730.log 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34570.log 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34730.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34650.log.gz 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-client.admin.34570.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34730.log.gz 2024-03-29T11:46:11.591 INFO:teuthology.orchestra.run.smithi203.stderr: -- /var/log/ceph/ceph.log 2024-03-29T11:46:11.592 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mgr.x.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34570.log.gz 2024-03-29T11:46:11.592 INFO:teuthology.orchestra.run.smithi203.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:46:11.592 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34690.log 2024-03-29T11:46:11.592 INFO:teuthology.orchestra.run.smithi203.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.34690.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:46:11.592 INFO:teuthology.orchestra.run.smithi203.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34690.log.gz 2024-03-29T11:46:11.592 INFO:teuthology.orchestra.run.smithi203.stderr: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:46:11.624 INFO:teuthology.orchestra.run.smithi145.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:46:22.860 INFO:teuthology.orchestra.run.smithi203.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:46:23.136 INFO:teuthology.orchestra.run.smithi203.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:46:23.236 INFO:teuthology.orchestra.run.smithi145.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:46:27.080 INFO:teuthology.orchestra.run.smithi203.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:46:27.246 INFO:teuthology.orchestra.run.smithi145.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:46:27.268 INFO:teuthology.orchestra.run.smithi203.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:46:27.294 INFO:teuthology.orchestra.run.smithi145.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:46:28.219 INFO:teuthology.orchestra.run.smithi145.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:46:59.030 INFO:teuthology.orchestra.run.smithi145.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:47:03.282 INFO:teuthology.orchestra.run.smithi145.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:47:03.283 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-03-29T11:47:03.284 INFO:teuthology.orchestra.run.smithi145.stderr:real 0m51.764s 2024-03-29T11:47:03.284 INFO:teuthology.orchestra.run.smithi145.stderr:user 2m33.375s 2024-03-29T11:47:03.284 INFO:teuthology.orchestra.run.smithi145.stderr:sys 0m5.457s 2024-03-29T11:47:19.781 INFO:teuthology.orchestra.run.smithi203.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:47:19.782 INFO:teuthology.orchestra.run.smithi203.stderr: 2024-03-29T11:47:19.783 INFO:teuthology.orchestra.run.smithi203.stderr:real 1m8.225s 2024-03-29T11:47:19.783 INFO:teuthology.orchestra.run.smithi203.stderr:user 1m57.814s 2024-03-29T11:47:19.783 INFO:teuthology.orchestra.run.smithi203.stderr:sys 0m4.240s 2024-03-29T11:47:19.783 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:47:19.783 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/remote/smithi145/log 2024-03-29T11:47:19.784 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:47:42.733 DEBUG:teuthology.misc:Transferring archived files from smithi203:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/remote/smithi203/log 2024-03-29T11:47:42.734 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:48:01.702 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:48:01.916 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=d430a97cebc547e6817d96d5517042d3 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:48:01.920 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:48:01.929 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:48:01.941 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:48:01.941 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:48:01.955 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:48:02.029 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:48:02.038 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:48:02.038 DEBUG:teuthology.orchestra.run.smithi145:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:48:02.041 DEBUG:teuthology.orchestra.run.smithi203:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:48:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:48:02.057 INFO:teuthology.orchestra.run.smithi203.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:48:02.059 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:48:02.059 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-03-29T11:48:02.060 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 53 +393us[ +387us] +/- 73ms 2024-03-29T11:48:02.060 INFO:teuthology.orchestra.run.smithi145.stdout:^* hv02.front.sepia.ceph.com 3 6 377 53 -132us[ -138us] +/- 68ms 2024-03-29T11:48:02.060 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 -237us[ -237us] +/- 67ms 2024-03-29T11:48:02.060 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:48:02.061 INFO:teuthology.orchestra.run.smithi203.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:48:02.062 INFO:teuthology.orchestra.run.smithi203.stdout:=============================================================================== 2024-03-29T11:48:02.062 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 54 +389us[ +388us] +/- 73ms 2024-03-29T11:48:02.062 INFO:teuthology.orchestra.run.smithi203.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -136us[ -137us] +/- 68ms 2024-03-29T11:48:02.062 INFO:teuthology.orchestra.run.smithi203.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 -244us[ -246us] +/- 67ms 2024-03-29T11:48:02.062 INFO:teuthology.orchestra.run.smithi203.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:48:02.063 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:48:02.072 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:48:02.073 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:48:02.081 DEBUG:teuthology.orchestra.run.smithi145:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:48:02.172 DEBUG:teuthology.orchestra.run.smithi203:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:48:02.244 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:48:02.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:48:02.277 DEBUG:teuthology.orchestra.run.smithi203:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:48:02.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:48:02.309 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:48:02.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:48:02.331 INFO:teuthology.task.internal:Duration was 1295.365679 seconds 2024-03-29T11:48:02.331 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:48:02.340 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:48:02.340 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:48:02.343 DEBUG:teuthology.orchestra.run.smithi203:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:48:02.385 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:48:02.396 INFO:teuthology.orchestra.run.smithi203.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:48:02.735 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:48:02.736 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2024-03-29T11:48:02.736 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:48:02.756 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi203.front.sepia.ceph.com 2024-03-29T11:48:02.756 DEBUG:teuthology.orchestra.run.smithi203:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:48:02.782 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:48:02.782 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:48:02.799 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:48:02.854 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:48:02.854 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:48:02.868 DEBUG:teuthology.orchestra.run.smithi203:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:48:02.996 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:48:03.006 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:48:03.006 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:48:03.033 DEBUG:teuthology.orchestra.run.smithi203:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:48:03.061 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:48:03.070 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:48:03.076 DEBUG:teuthology.orchestra.run.smithi203:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:48:03.099 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2024-03-29T11:48:03.127 INFO:teuthology.orchestra.run.smithi203.stdout:kernel.core_pattern = core 2024-03-29T11:48:03.167 DEBUG:teuthology.orchestra.run.smithi145:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:48:03.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:48:03.181 DEBUG:teuthology.orchestra.run.smithi203:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:48:03.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:48:03.225 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:48:03.343 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:48:03.345 DEBUG:teuthology.misc:Transferring archived files from smithi145:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/remote/smithi145 2024-03-29T11:48:03.345 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:48:03.389 DEBUG:teuthology.misc:Transferring archived files from smithi203:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713/remote/smithi203 2024-03-29T11:48:03.390 DEBUG:teuthology.orchestra.run.smithi203:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:48:03.432 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:48:03.433 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:48:03.436 DEBUG:teuthology.orchestra.run.smithi203:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:48:03.477 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:48:03.492 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:48:03.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:48:03.523 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:48:03.523 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:48:03.526 DEBUG:teuthology.orchestra.run.smithi203:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:48:03.538 INFO:teuthology.orchestra.run.smithi145.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:48 /home/ubuntu/cephtest 2024-03-29T11:48:03.541 INFO:teuthology.orchestra.run.smithi203.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:48 /home/ubuntu/cephtest 2024-03-29T11:48:03.542 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:48:03.552 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:48:03.597 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:48:03.619 DEBUG:teuthology.nuke:shortname: smithi145 2024-03-29T11:48:03.619 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:48:03.642 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:38.098177', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPiPMOp+vgiEZZNa8LPYohHLQRxJY9R6LGA1gZHjclHL8fx7cA2Mxw4cYFT8DwlUftEIvyJowLK2lvMRhM8DeK8='} 2024-03-29T11:48:03.647 DEBUG:teuthology.nuke:shortname: smithi203 2024-03-29T11:48:03.647 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:48:03.667 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi203.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622713', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:16:38.100132', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEZDVBhzAjM1D8MQiaC92e7nFlm/phBSL2LVW0XsOysQ2+y1EDEUrkFPDQ5iXfuu8mbjppQok81+OL7WRQ6WzLs='} 2024-03-29T11:48:03.700 INFO:teuthology.orchestra.console.smithi145:Power off 2024-03-29T11:48:03.701 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:48:03.738 INFO:teuthology.orchestra.console.smithi203:Power off 2024-03-29T11:48:03.739 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:48:03.752 DEBUG:teuthology.orchestra.console.smithi145:power off output: Chassis Power Control: Down/Off 2024-03-29T11:48:03.753 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:03.768 DEBUG:teuthology.orchestra.console.smithi203:power off output: Chassis Power Control: Down/Off 2024-03-29T11:48:03.768 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:03.788 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-29T11:48:03.793 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-29T11:48:07.789 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:07.799 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:07.902 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-29T11:48:07.909 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-29T11:48:11.907 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:11.916 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:12.018 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-03-29T11:48:12.028 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is on 2024-03-29T11:48:16.022 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:16.032 DEBUG:teuthology.orchestra.console.smithi203:pexpect command: ipmitool -H smithi203.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:48:16.134 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-03-29T11:48:16.142 DEBUG:teuthology.orchestra.console.smithi203:check power output: Chassis Power is off 2024-03-29T11:48:16.235 INFO:teuthology.orchestra.console.smithi145:Power off completed 2024-03-29T11:48:16.243 INFO:teuthology.orchestra.console.smithi203:Power off completed 2024-03-29T11:48:16.347 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1295.3656785488129 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=d430a97cebc547e6817d96d5517042d3 status: fail success: false 2024-03-29T11:48:16.348 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:48:16.421 INFO:teuthology.run:FAIL